Foreign Policy

dvmc · 09-25-2012, 12:58 PM

Hi there guys.

I was wondering if anyone was experiencing problems with FP? It was working until a few months ago, now all that gets downloaded is a set of html links to the articles or a table of contents.

This is both in the viewer on the computer and on my Kindle.

I'm using Calibre on linux.

Thanks for any help.

kiklop74 · 09-25-2012, 04:41 PM

I'll take a look

kiklop74 · 09-25-2012, 05:17 PM

The seem to have some nasty javascript protection against scraping. Too much work for me.

Steven630 · 09-28-2012, 09:37 PM

Quote:

Originally Posted by kiklop74

The seem to have some nasty javascript protection against scraping. Too much work for me.

All section names are images, not texts. This seems to be the biggest problem. Besides, the structure is such that every single article looks like a section to me. But isn't it possible to fetch all articles without putting them into various sections (like what you do when you fetch articles from a rss)?

kiklop74 · 09-29-2012, 06:45 AM

The problem is not in fetching urls it is in fetching the content of articles. The content is being loaded by javascript after the main page loads.

Steven630 · 10-01-2012, 04:54 AM

Quote:

Originally Posted by kiklop74

The problem is not in fetching urls it is in fetching the content of articles. The content is being loaded by javascript after the main page loads.

Indeed. Thank you. It used to be fine. What a pity.

ee7711 · 10-01-2012, 06:51 AM

I'm afraid to be out of place here, but it seems work:

class AdvancedUserRecipe1349086293(BasicNewsRecipe):
title = u'Foreign Policy'
oldest_article = 7
max_articles_per_feed = 100
auto_cleanup = True

feeds = [(u'Foreign_Policy', u'http://www.foreignpolicy.com/node/feed')]

def print_version(self, url):
return url + '?print=yes&hidecomments=yes&page=full'

Steven630 · 10-07-2012, 09:43 AM

It has nothing to do with the recipe. The website has been changed, it seems.

Steven630 · 10-07-2012, 09:45 AM

Now that the recipe works again, can anyone make a recipe of the print edition?
http://www.foreignpolicy.com/issues/current

09-25-2012, 12:58 PM	#1
dvmc Junior Member Posts: 1 Karma: 10 Join Date: Sep 2012 Device: Kindle	Foreign Policy Hi there guys. I was wondering if anyone was experiencing problems with FP? It was working until a few months ago, now all that gets downloaded is a set of html links to the articles or a table of contents. This is both in the viewer on the computer and on my Kindle. I'm using Calibre on linux. Thanks for any help.

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Free (nook/Kindle/Sony/iTunes) Foreign Policy [Globetrotting Insurance Agent Mystery]	ATDrake	Deals and Resources (No Self-Promotion or Affiliate Links)	2	09-01-2012 10:26 AM
Foreign Policy all feeds? Help?	DogonCrook	Recipes	1	11-10-2010 06:44 PM
Classic Return Policy	nremondelli	Barnes & Noble NOOK	1	06-10-2010 09:31 AM
Return Policy	JicMic	Sony Reader	1	04-25-2009 10:37 AM

09-25-2012, 04:41 PM	#2
kiklop74 Guru Posts: 800 Karma: 194644 Join Date: Dec 2007 Location: Argentina Device: Kindle Voyage	I'll take a look

09-25-2012, 05:17 PM	#3
kiklop74 Guru Posts: 800 Karma: 194644 Join Date: Dec 2007 Location: Argentina Device: Kindle Voyage	The seem to have some nasty javascript protection against scraping. Too much work for me.

09-29-2012, 06:45 AM	#5
kiklop74 Guru Posts: 800 Karma: 194644 Join Date: Dec 2007 Location: Argentina Device: Kindle Voyage	The problem is not in fetching urls it is in fetching the content of articles. The content is being loaded by javascript after the main page loads.

10-01-2012, 06:51 AM	#7
ee7711 Junior Member Posts: 1 Karma: 10 Join Date: Oct 2012 Device: kindle	I'm afraid to be out of place here, but it seems work: class AdvancedUserRecipe1349086293(BasicNewsRecipe): title = u'Foreign Policy' oldest_article = 7 max_articles_per_feed = 100 auto_cleanup = True feeds = [(u'Foreign_Policy', u'http://www.foreignpolicy.com/node/feed')] def print_version(self, url): return url + '?print=yes&hidecomments=yes&page=full'

10-07-2012, 09:43 AM	#8
Steven630 Groupie Posts: 154 Karma: 10 Join Date: May 2012 Device: Kindle Paperwhite2	It has nothing to do with the recipe. The website has been changed, it seems.

10-07-2012, 09:45 AM	#9
Steven630 Groupie Posts: 154 Karma: 10 Join Date: May 2012 Device: Kindle Paperwhite2	Now that the recipe works again, can anyone make a recipe of the print edition? http://www.foreignpolicy.com/issues/current