Occasional Garbled Characters using this recipe

[Bullpeeper]

Using a Kindle 3 and latest version of Calibre. The attached recipe for The Columbus Dispatch returns several articles from RSS feeds with funky characters, especially lots of squares. I've noticed several others with similar problems with different feeds and it seems there might not be an answer - hoping that one of you with .py experience would at least look at the attached recipe i downloaded from the archive to see if there's an issue with it. I have also tried to use the basic mode and just let Calibre create the recipe after i add feeds individually -- those download as messages about flash, the article text, but everything is justified center. Thanks for any guidance anyone may have. I'm not a programmer, so any help is appreciated.

[desertgrandma]

Welcome to MobileRead, Bullpeeper

Someone should be along soon with some suggestions.

Enjoy the forums.

[Starson17]

Originally Posted by Bullpeeper

Using a Kindle 3 and latest version of Calibre. The attached recipe for The Columbus Dispatch returns several articles from RSS feeds with funky characters, especially lots of squares.

Do the "squares" appear only on the Kindle? Do you see them in the Calibre viewer?

[Bullpeeper]

Thanks, Starson17. The corrupt articles don't have squares in the Calibre viewer but are garbled text. Here's a sample:

������_����K���XIďż˝8ďż˝ k;@��������.ďż˝H3I��4Zďż˝WVVÇ �� ďż˝?+2ďż˝z+���utux�����Yďż˝ď ż˝[��yt{ďż˝sz{qÎŞďż˝ �ڑ�1ďż˝e���Q��c��²��Y=ďż˝ď ż˝ďż˝lEV�� ďż˝qďż˝A9ďż˝o����fďż˝'��w_��Vďż˝ďż ˝6ďż˝$ďż˝/���x��*?���Hďż˝ďż˝ďż˝ďż˝ďż˝ďż˝ďż˝ď ż˝ďż˝ŮĄĎŁďż˝Q����E|2ďż˝a�����8ďż ˝ďż˝ďż˝ďż˝ďż˝ďż˝ďż˝ďż˝l ��Jďż˝yďż˝H����!~����qďż˝ďż˝ďż ˝ďż˝(����꼳V]X��ڸ��E&ďż˝ y��n�������Zďż˝|$����� 5���&{��

[Starson17]

Originally Posted by Bullpeeper

Thanks, Starson17. The corrupt articles don't have squares in the Calibre viewer but are garbled text.

If you fetch the recipe several times, does the same article always have garbled text, or is it sometimes OK and another has problems?

[Bullpeeper]

I've noticed that a particular article will have garbled text one day and if that article happens to come down in the next day's feed as well, sometimes the text will be fine. I would say that it's different articles, but a day has to go by. If i fetch the same recipe in the same day, the same article will have garbled text ... thanks.

[Starson17]

Originally Posted by Bullpeeper

I've noticed that a particular article will have garbled text one day and if that article happens to come down in the next day's feed as well, sometimes the text will be fine. I would say that it's different articles, but a day has to go by. If i fetch the same recipe in the same day, the same article will have garbled text ... thanks.

It sounds like they may be putting ads or something else into the feeds that interfere. You'll need to look closely manually to see if you can spot the difference. Once you know what it is, you may be able to remove it in the recipe.

[Bullpeeper]

Got it, and thanks! When i use Calibre's basic method for creating a news source, by just pasting in the Dispatch's rss link, what fetching produces comes out as all of the text, but justified center. There are also java script and/or flash messages which seem to have something to do with ads at the beginning of many articles. If there are anti-ad recipe elements you know of, please post or link. I'm not a programmer but I'm picking up how some of this works just by comparing recipes that work well with mine that don't.

[Starson17]

I finally had time to look at this.
Most of your links from the RSS feed work normally, but occasionally a link like this:

http://www.dispatch.com/live/content/local_news/stories/2010/10/24/an-illusion-of-treatment.html?sid=101

is sent off to a link like this:

http://www.dispatchpolitics.com/live/content/local_news/stories/2010/10/24/copy/an-illusion-of-treatment.html?adsec=politics&sid=101

resulting in the problems you see. (Don't you love the political ad season!)
This is done by adding a <meta http-equiv="refresh"> tag to the code.
Add this to your recipe to prevent it:

remove_tags.append(dict(name = 'meta', attrs = {'http-equiv': 'refresh'}))

[kovidgoyal]

Another way to do it is with

def get_browser(self):
   br = BasicNewsRecipe.get_browser(self)
   br.set_handle_refresh(False)
   return br

[Starson17]

Originally Posted by kovidgoyal

Another way to do it is

Thanks! I didn't think of that option. For my personal reference, as I sometimes search here for bits of code I recall but can't find, there are other optional settings for Mechanize and a list can be found here:
http://wwwsearch.sourceforge.net/mechanize/

[Bullpeeper]

Originally Posted by kovidgoyal

Another way to do it is with

def get_browser(self):
   br = BasicNewsRecipe.get_browser(self)
   br.set_handle_refresh(False)
   return br

Thanks -- and thanks for the creation of this software.

[Bullpeeper]

Originally Posted by Starson17

I finally had time to look at this.
Most of your links from the RSS feed work normally, but occasionally a link like this:

http://www.dispatch.com/live/content/local_news/stories/2010/10/24/an-illusion-of-treatment.html?sid=101

is sent off to a link like this:

http://www.dispatchpolitics.com/live/content/local_news/stories/2010/10/24/copy/an-illusion-of-treatment.html?adsec=politics&sid=101

resulting in the problems you see. (Don't you love the political ad season!)
This is done by adding a <meta http-equiv="refresh"> tag to the code.
Add this to your recipe to prevent it:

remove_tags.append(dict(name = 'meta', attrs = {'http-equiv': 'refresh'}))

Starson17 - Greatly appreciate you taking a further look at this. I'm going to cut and paste your code or the code offered by kovidgoyal. This Calibre thing and the recipes for news sources is my favorite thing about my Kindle lately ...