Politifact Recipe Problems

[grizliez]

I'm having problems with the Politifact recipe. It posts the short descriptions fine in the section listing, but some of the actual articles are just a mess of symbols and special characters. Many of the articles come out fine. I've tried figuring out what is being interpreted in the python code that is causing this so that I can set the code to remove the offending tags, but without success. I'm new at this so I'm probably missing something. Ideas?

[Starson17]

Originally Posted by grizliez

Ideas?

I've seen this when ads are being randomly inserted. See if that's happening. I've also seen it when redirects occur and the processing isn't following quickly enough. Try adding a delay and running a single thread download:

             simultaneous_downloads = 1
             delay = 5

[grizliez]

Originally Posted by Starson17

I've seen this when ads are being randomly inserted. See if that's happening. I've also seen it when redirects occur and the processing isn't following quickly enough. Try adding a delay and running a single thread download:

             simultaneous_downloads = 1
             delay = 5

I gave the suggestion a try and it didn't work. It does appear to be happening for the same stories each time, so it is repeatable. Still not sure what's going on.

[grizliez]

Just found the problem after searching further and following one of the ideas in the reusable code section. It was all about the links in certain stories. I used the code segment that converts links to text and now no more problems. Code reprinted here for the next person

Spoiler:

def preprocess_html(self, soup):
for alink in soup.findAll('a'):
if alink.string is not None:
tstr = alink.string
alink.replaceWith(tstr)
return soup

[KNickerson]

Is there a way to get this into the Calibre release? I'm seeing the same issue with the latest version.

[Starson17]

Originally Posted by KNickerson

Is there a way to get this into the Calibre release? I'm seeing the same issue with the latest version.

The "fix" posted isn't designed to be a fix for this problem, even though it seems to have solved it in one case in the past. The code is designed to remove clickable links, while leaving the text. Many devices can use the clickable links, so that isn't a desirable solution in many cases.

I checked it out by adding that code to the Politifact recipe , and it doesn't solve the problem.

[KNickerson]

Originally Posted by Starson17

I checked it out by adding that code to the Politifact recipe , and it doesn't solve the problem.

(Disclaimer: This is the first I've looked into the recipes).

I'm thinking it's a download problem. I copied the script off and ran
ebook-convert PolitifactKJN.recipe .epub -vv --debug-pipeline debug

Then I found a bad section, and hunted it down in debug\input.
The index.html there is just garbage. Isn't that the raw stuff downloaded before the recipe kicks in? If not, how do I get to the raw stuff?

[Starson17]

Originally Posted by KNickerson

(Disclaimer: This is the first I've looked into the recipes).

I'm thinking it's a download problem. I copied the script off and ran
ebook-convert PolitifactKJN.recipe .epub -vv --debug-pipeline debug

Then I found a bad section, and hunted it down in debug\input.
The index.html there is just garbage. Isn't that the raw stuff downloaded before the recipe kicks in? If not, how do I get to the raw stuff?

That should be the raw stuff, but to be sure you can do this:

    def preprocess_html(self, soup):
        print 'The raw stuff is: ', soup  
        return soup

Is it always the same crud at the same point?

[KNickerson]

Originally Posted by Starson17

Is it always the same crud at the same point?

It's not clear. Yesterday, I was looking at a check on Krugman, and in several runs it was always bad, but then it was ok. Today that one is still ok, but I've had four runs where a check on farm tractors is garbage.

It is the raw data though, the debug snippet you gave me shows the crud. I also see that all of the crud shows "WARNING: Encoding detection confidence 0%"

I captured the complete fetch with WireShark, and I can't find any garbage in the capture. I did find at least one reply that came in gzip'd though, I don't know if Calibre can handle a gzip'd response.

[kovidgoyal]

A server should never send gzip if the client doesn;t say it accepts it. But you can add gzip support to a particular recipe by adding:

def get_browser(self):
   br = BasicNewsRecipe.get_browser(self)
   br.set_handle_gzip(True)
   return br

[KNickerson]

Originally Posted by kovidgoyal

A server should never send gzip if the client doesn;t say it accepts it. But you can add gzip support to a particular recipe by adding:

def get_browser(self):
   br = BasicNewsRecipe.get_browser(self)
   br.set_handle_gzip(True)
   return br

That looks good. I added it in, and got a perfect epub output. Granted, given the variation it might be just luck, but I'm hopeful.

Given that gzip is possible, is there any reason to not decode gzip even if it wasn't requested?

[Starson17]

If it continues to work, give us an update. I can't recall seeing anything quite like this before, but it's a handy tool to know about.

[KNickerson]

Originally Posted by Starson17

If it continues to work, give us an update. I can't recall seeing anything quite like this before, but it's a handy tool to know about.

I'm totally convinced this is the fix. I reverted back to the original recipe, and did another packet capture with WireShark. I searched that capture for gzipped data. There was a perfect one-to-one match: All gzipped data was garbage in the epub, and all garbage in the epub was gzipped data.

I also see that the Obamameter feed isn't right, it needs to somehow follow another link in, but that's not a very interesting feed to me.

Anyone know what procedural hoops I need to go through to get this into the official release? (Yes, I should just RTM)

[Starson17]

Originally Posted by KNickerson

Anyone know what procedural hoops I need to go through to get this into the official release? (Yes, I should just RTM)

Kovid will usually pick it up here. He probably prefers a complete tested recipe, rather than a code chunk to add in, which may require more testing from him, but this one's pretty simple.

[kovidgoyal]

This is already in 0.8.16