Certain PNG files fail to load

[RobFreundlich]

Certain PNG files in one of my feeds are failing to load, with the following error:

Fetching http://c.o0bg.com/rw/Boston/2011-2020/2012/09/25/Comics/aj_c120925.png
Traceback (most recent call last):
  File "site-packages\calibre\web\fetch\simple.py", line 388, in process_images
  File "site-packages\PIL\Image.py", line 1982, in open
IOError: cannot identify image file

Unlike the last time I posted something like this, where it was my own mistake, this seems to be a problem in the image files themselves. The image file at http://c.o0bg.com/rw/Boston/2011-202...aj_c120925.png seems to have something weird about it - I can open it in Chrome and Firefox (or, for that matter, GIMP), but IE 8 gives me a "red X broken image" picture for it.

I've attached a copy here as well. This simple script fails with the same error message as calibre (not surprising, given that it uses PIL):

from PIL import Image

im = Image.open("aj_c120925.png")
print "Format:", im.format, ", Mode:", im.mode, ", Size:", im.size

Can anyone offer any advice on this?

[kovidgoyal]

There isn't anything simple you can do about it, but if you are willing to code a little you can impleent image_url_processor in your recipe. There you can download the image and try processing ti with imagemagick (which calibre bundles). Maybe imagemagick can "fix" whatever is wrong with the image and then return the url of the fixed image so that PIL will not choke.

[RobFreundlich]

Thanks - I'll give that a try. One more question: I looked in the pylib.zip that came with Calibre, and don't see imagemagick. Where is it bundled, and what's the correct import statement?

[kovidgoyal]

See utils/magick/draw.py

[RobFreundlich]

Thanks to your tips, I've solved this problem. Here's my solution:

    valid_filename_chars = "-_.%s%s" % (string.ascii_letters, string.digits)

    def image_url_processor(self, baseurl, url):
        self.log("===================\nbaseurl: ", baseurl, "\nurl: ", url)
        # This is a hack because some of the URLs just have a leading
        # // instead of http://
        if url.startswith("//"):
            url = "http:" + url

        url = self.get_image(url)

        self.log("url out: ", url, "\n===================")

        return url

    def get_image(self, url):
        # Another hack - sometimes the URLs just have a leading /,
        # in which case I stick on "http://" and the correct domain
        if url.startswith("/"):
          url = self.make_url(url)

        # Get the image bytes
        br = BasicNewsRecipe.get_browser()
        response = br.open(url)
        data = response.get_data()

        # write it to a local file whose name is based on the URL
        filename = ''.join(c for c in url if c in self.valid_filename_chars)
        self.log("filename=%s" % filename)

        f = open(filename, "wb")
        f.write(data)
        f.close()

        # Try to read it with PIL, which is what the containing app will do
        try:
            im = PIL.Image.open(filename)
        except:
            # If it failed, read it with ImageMagick and write it to a new file,
            # changing the URL to point to the new file
            self.log("Could not open ", filename, " from ", url)
            self.log("Trying to open and re-save with ImageMagick")
            image = calibre.utils.magick.Image()
            image.read(filename)
            image.save("new_" + filename)
            url = os.getcwd() + "/new_" + filename
            url = "file:///" + url.replace("\\", "/")
            self.log("Succeeded.  Using local file")

        return url

Luckily, ImageMagick manages to load the file successfully AND heal it when saving it back out, so I didn't have to look into what exactly was wrong with these files. For curiosity's sake, I did do a comparison of the old and new, and there are differences, but since I don't know squat about the PNG format (and don't have the time or energy to learn), I don't know exactly what they mean.

If I had the time, I'd grab the calibre source, find where it's doing the image load, put something like this in, and submit it as a bug fix, but unfortunately, I don't. If anyone else out there wants to do it, go ahead - I happily release this code (particularly the try...except bit that solves the problem) into the public domain.