Calibre Globe and Mail recipe and Sony PRS-600

[elvenic]

When I download Globe and Mail feed using Calibre recipe and then download it to my Sony PRS-600 and then view it, there is always a page on which the PRS-600 freezes for very long time, even 'reset' does not help immediately, only after couple of minutes of being frozen if I try to repeat 'reset' and power-on the device would reset and come back alive.

I tried to modify the recipe and use if from the command line (ebook-convert) and look at the downloaded HTML to figure out what exactly in the Globe and Mail HTML freezes the PRS-600, but without success.

Any help with this?

Also, when I download generated epub to the Sony PRS-600, Calibre reports following error:

ERROR: ERROR: Unhandled exception: <b>AttributeError</b>:'NoneType' object has no attribute 'split'

Traceback (most recent call last):
File "/home/kovid/build/calibre/src/calibre/gui2/__init__.py", line 245, in dispatch
File "/home/kovid/build/calibre/src/calibre/gui2/device.py", line 902, in books_uploaded
File "/home/kovid/build/calibre/src/calibre/gui2/device.py", line 258, in add_books_to_metadata
File "/home/kovid/build/calibre/src/calibre/devices/prs505/driver.py", line 168, in add_books_to_metadata
AttributeError: 'NoneType' object has no attribute 'split'

[kovidgoyal]

Regrading the error. Did you customize the sony reader and remove all fields for creating collections?

If you can post a link to/attach the html that causes problems, I may have a few suggestions

[elvenic]

Originally Posted by kovidgoyal

Regrading the error. Did you customize the sony reader and remove all fields for creating collections?

No, I did not do anything to the Sony Reader, and I don't even know what are "fields for creating collections"

Originally Posted by kovidgoyal

If you can post a link to/attach the html that causes problems, I may have a few suggestions

The 'link' is in the embedded 'Globe and Mail' recipe: just use 'download now', it will generate Globe and Mail epub in the Calibre's list of books; then if you connect Sony PRS-600 to the computer via USB, Calible will download the epub to the Sony reader. Disconnect the USB, when Sony wakes up from the USB stupor, open the Globe and Mail book on it and start flipping pages. You may flip through couple of first articles without a problem, but very soon - this always happens to me within first 50 pages - when you press 'next page', the Sony freezes and I have to do 'reset' to bring it back to life (well, may be if I waited another 10 min. the Sony would wake up by itself)

I think that today at night, when I get home, I might be able to come up with a small epub file that causes the reader to freeze: I was playing with the contents of the directory that

ebook-convert globe_and_mail.recipe <dir>

created and was building the epubs from it with

ebook-convert <dir>/index.html ttt.epub

and downloading them to the Sony PRS-600 via the Calibre - I'm just trying to find out which exactly generated HTML page causes the Sony to freeze.

[kovidgoyal]

Use

ebook-convert globe_and_mail.recipe .epub -vv --debug-pipeline p --extract-to x

html from various stages of the download and conversion process will be saved in p and the epub will be unzipped into x

[elvenic]

OK, first strange results.

I have attached two files:

ttt.tar.gz - contains a directory 'ttt' that contains a file hierarchy with html and other files from which a ebub is built,

ttt.epub - the epub that was built from the directory 'ttt'

In the directory 'ttt', in the subdirectory 'feed_0', only first two subdirectories ('article_1' and 'article_2') contain index.html files that came from the Globe and Mail feed. All other 'article_*' subdirectories contain dummy index.html files with a simple html text 'Empty Article' (in all these directories, the original index.html files frrom the G&M feed are saved under names 'index.html.saved', so that you can restore the real feed data. I don't think that files with extension '.saved' that are not referenced anywhere in the .html files have any influence on the contents of the resulting epub file.)

The _expected_ result would be that the epub file, when viewed, would show first two articles as from the real G&M feed, and third and subsequent articles with text "Empty Article".

The _observed_ result is that when viewed, the epub file shows first two articles as expected, with the text from the real G&M feed, but third and subsequent articles are shown all as the text of the first article (may be there is a recursion somewhere in the feed?) . This happens both when viewed by the Calibre internal e-book viewer, and when downloaded to the Sony PRS-600. But the Sony does not freeze in this case.

This is a strange result - epub should not contain multiple copies of the first article instead of the articles with "Empty Article" text. May be this is a manifestation of the same error that causes the Sony to freeze on the bigger epub?

[kovidgoyal]

Check the <spine> and <manifest> elements in the opf file that will tell you what files are being read.

[elvenic]

Originally Posted by kovidgoyal

Check the <spine> and <manifest> elements in the opf file that will tell you what files are being read.

<spine> and <manifest> point to all the right files: when I removed original 'index.html' files, I substituted them with dummy 'index.html' files that contain syntactically correct html. So everything that <manifest> refers to is still there.

But I have just made another discovery : the original index.html file in the third article directory ('article_2') subdirectory contains strange <img> element (with empty 'src' attribute):

<img src="" width="" height="" alt="" class="calibre15"/>

When this element is present in the epub, Sony freezes when it tries to show third article.

When I commented this out :

<!-- <img src="" width="" height="" alt="" class="calibre15"/> -->

and build epub and download it to the Sony, it did not freeze! Maybe this is a source of this trouble?

Is there a way to make the recipe to ignore <img> elements with empty src?

Or may be this should be hardcoded somewhere in the Calibre source as a basic sanity check: what is a point of an image with no src URL?

[kovidgoyal]

Sigh! Seriously who wrote the code for adobe digital editions? I'll add a workaround in the next release.

[elvenic]

Unfortunately, that's not all of it.

I added more download feeds to the recipe that I was playing with, trying to confirm that we have found the source of the bug. I found a img with empty src - there was only one in all download, and removed this line. Then I build the pub (now it has 250 pages) and downloaded it to the Sony device.

The page with the image that I removed shown OK.

But on the next page Sony froze.

The next page contained several images that were setup as a background images with URLs provided in a stylesheet file, and in the stylesheet file the URLs of these images were pointing to the real web addresses of the Globe and Mail servers.

For example, the index.html file in the article_4 directory contains following link:

<a href="http://www.theglobeandmail.com/video/teen-girl-violence-on-rise/article1430931/?view=picks" class="calibre47">Teen girl violence on rise</a>

(notice class="calibre47" attribute).

The stylesheet file, ../..stylesheet.css, contains following regarding class 'calibre47':

.calibre47 {
    background-image: url(http://beta.images.theglobeandmail.com/archive/00433/girl_fight_poll__433794gm-d.jpg)
    }

As you can see, the URL points to the "beta.images.theglobeandmail.com", that Sony PRS-600 can't access because it does not have internet connectivity.

I'm not sure how Calibre processes CSS, but may be it would be possible to filter out all external URLs from it when downloading the feed?

Would it be possible to achieve this by adding some code to the recipe? Any advice?

[elvenic]

Yes, I confirm that the URLs in the "background-image:" stylesheet element that point to external website were a reeson for the second freese: when I manually removed the link to the ../../stylesheet.css file from the index.html files that wanted to show these background images, build epub and downloaded it to the Sony, I was able to flip through all 250 pages of it without the device freezing.

[elvenic]

Hm, how do I print debug information from postprocess_html()?

And, even more important question, would it be possible to remove a link to the '../../stylesheet.css' file by overriding postprocess_html()? Or the link to this stylesheet file is generated by the Calible after the postprocess_html() returns?

[kovidgoyal]

I just removed the video articles from the globe and mail recipe, that should fix this automatically.

[elvenic]

Originally Posted by kovidgoyal

I just removed the video articles from the globe and mail recipe, that should fix this automatically.

OK, where can I download the new recipe from? I would like to try it with my Sony PRS-600.

And, the original embedded G&M recipe that I started playing with from was removing ALL 'img' tags from the html - i modified it not to remove images, and other then the problems that I described above (empty 'src' and background images via css, but that is not really about 'img' tag) I did not have any troubles. And having images in the Globe and Mail articles greatly improves reading experience

[kovidgoyal]

If you use the builtin recipe calibre should automatically use the updated recipe. (provided your version of calibre > 0.6.29)