Incomplete magazines (Smithsonian, Scientific American)

[fontCrazee]

i apologize if this question has been asked before: lately i'm not receiving many articles or any articles in the magazines i download using Calibre. specifically Smithsonian Mag, The New Yorker, Psychology Today and Scientific American. Are these no longer offered for free?
Thanks.

[kovidgoyal]

Many news websites lately are moving to using paywalls. Visit the website and check foryourself. It may just be that the website has changed and the recipe needs adjusting in calibre.

[kiklop74]

None of the mentioned magazines ever offered full content for free. They are just reducing the number of freely acailable articles.

[nhow]

Originally Posted by kovidgoyal

Many news websites lately are moving to using paywalls. Visit the website and check foryourself. It may just be that the website has changed and the recipe needs adjusting in calibre.

I'm having the same problem with the New Yorker. Calibre downloads the online version but many articles do not show the text only a link to the online edition (where, if you click on it you can see the whole article). I checked the current recipe and it includes the code for showing articles where there's a link. I think that's what this is:

def preprocess_html(self, soup):
for item in soup.findAll(style=True):
del item['style']
auth = soup.find(attrs={'id':'articleauthor'})
if auth:
alink = auth.find('a')
if alink and alink.string is not None:
txt = alink.string
alink.replaceWith(txt)
return soup

I'm not very comfortable editing code but would try it if there's something new that would work.

Many thanks for your help.

[kiklop74]

Originally Posted by nhow

def preprocess_html(self, soup):

This part of code does not download any additional content. I'm just not interested in updating the recipe. Too much work for free.

[rainrdx]

With the exception of Smithsonian, all other three publish an increasingly smaller number of their articles online (New Yorker publishes their articles in a flash-based archive, I believe, which makes making a recipe incredibly hard, if not theoretically impossible). Unless they change their policy, there is not much can be done, I think.

[nhow]

Originally Posted by kiklop74

This part of code does not download any additional content. I'm just not interested in updating the recipe. Too much work for free.

I can understand that. I'm really grateful to you and all who have built and maintained this software. So I'll give another contribution.

Thank you.

[oneillpt]

Scientific American still provides content, but the recipe needs a fix:

Replace

for a in soup.find(attrs={'class':'primaryCol'}).findAll('a',attrs={'title':'Feature'}):

by

for a in soup.find(attrs={'class':'doubleWide'}).find(attrs={'class':'primaryCol'}).findAll('a',attrs={'title':'Feature'}):

This should be sufficient to get the recipe working again. If you wish you can also clean up articles a little more by replacing the remove_tags by:

remove_tags = [dict(name='a', attrs={'class':'tinyCommentCount'})
                                ,dict(name='div', attrs={'id':'bigCoverModule'})
                                ,dict(name='div', attrs={'class':'addInfo'})
                                ]