HOWTO: Improve performance on calibe generated ePUBs

[=X=]

Hi All,
NOTE1: Perl script with FIX is now added
NOTE2: Added executable! Thank you nrapallo!

Note: I decided to make my post #517 its on thread here in the SONY section

I've found the TOC on ePUB generated by calibre to be intolerable. An ePUB with forty TOC entry can take up to 90sec.

Below is what I've found


TOC with "#HREF" syntax makes opening the ePUB extremely slow. With large enough TOC files this will take a long time or even cause the reader to crash.


PROBLEM:
I've noticed a big performance hit every time I try to open up an ePUB book and use the TOC. You mentioned on a different thread it was due to the #HERF.

TEST:
Okay I've done a few test to see how true this is and if there is a good solution to resolve this.

Attached is 3 files
Test File.epub (unmodified calibre generated TOC)
Test File_NOREF.epub (ALL #HREF removed from all URL in the toc.ncx file)
Test File_noREF_Capter.epub (Only the top level chapters have the #HREF removed, sub chapters have the #HREF)

Measured time to the TOC from an ePUB book created from calibre.

• Test File.epub
  : 110 sec (1min 50 sec)
• Test File_NOREF.epub
  (Instant)
• Test File_noREF_Capter.epub
  : Instant for top level chapters. Sub chapters varied depending on how many sub elements it had. The last chapter had 40 items and took 1.5 sec

SOLUTION
There is a HUGE performance increase by just removing the the #HREF URL path from top level TOC. While there still is a hit on sub toc they are small and tolerable.

To do this unzip the epub. Open the toc.ncx XML file.

Go to the docTitle section
Then move to the childe node titled docTitle/navPoint/content XPath
<docTitle>
<navPoint>
<content src="URL">

Remove the #HREF portion located in the URL text of the content node. (i.e. at the end of the URL there is something "http://....#calibre_..." Remove everything from the hash (#) to the end of the URL.

This only has to be done for the top level navPoints to increase the performance.

Have Fun,

=X=

[=X=]

I started thinking this can easily be done with a perl script.

Is there any interested in such a script?

=X=

[JSWolf]

I'd be more interested in a Python script. But I would be interested in such a fix.

[=X=]

Hi Jon,

Unfortunately don't know how to program in Python. I can manage my way through existing code but writing code from scratch is a different story.

I'm almost complete with the perl script, just polishing it up. It should be ready by tonight or tomorrow morning.

=X=

[JSWolf]

Originally Posted by =X=

Hi Jon,

Unfortunately don't know how to program in Python. I can manage my way through existing code but writing code from scratch is a different story.

I'm almost complete with the perl script, just polishing it up. It should be ready by tonight or tomorrow morning.

=X=

Can you add a very needed feature to your script please? What's needed is in a multi-level ToC, the top level needs to be the first entry when the top level is opened as there is no way on the 505 or 700 to actually use the top level as a ToC entry. So if it's also not in the second level, it'll never be as a ToC entry.

[Ankh]

ADE on Sony is really sensitive to the size of the "chunk", which is the reason that I always-always-always hand-edit the results of the conversion to have a single file per TOC entry. The smallest possible size of the xhtml chunk really helps.

[=X=]

Originally Posted by JSWolf

Can you add a very needed feature to your script please? What's needed is in a multi-level ToC, the top level needs to be the first entry when the top level is opened as there is no way on the 505 or 700 to actually use the top level as a ToC entry. So if it's also not in the second level, it'll never be as a ToC entry.

Yea I will, that's a good idea. I'll have to look into this a little more though.

I used the Expat XML parser, one of the first XML parsers out there. It's easy to use but more importantly usually included in all Perl distributions. What I'm getting at is this does not support DOM so it's much harder to handle moving,copying nodes as a result. I'll look into it though it shouldn't be too hard.

=X=

[=X=]

Okay here is my first pass

ePub_TOC_enhancer.pl eBookName.epub

It also has a recursive mode just put the '-R' switch after the executable to update all ePubs in the current directory and all it's sub directories.

  usage: ePub_TOC_enhancer.pl eBookName.epub

        -h          : this (help) message
        -R          : Recursively search for ePUBs

  Example: ePub_TOC_enhancer.pl -R 
    Will recursively search for ePUBs in
    the current directory and all its children.

  Example: ePub_TOC_enhancer.pl MYeBook.epub YOUReBook.epub OUReBOOK.epub
    Will only fix the eBooks specified.

[=X=]

Originally Posted by Ankh

ADE on Sony is really sensitive to the size of the "chunk", which is the reason that I always-always-always hand-edit the results of the conversion to have a single file per TOC entry. The smallest possible size of the xhtml chunk really helps.

Hi Ankh,
What my first post is getting at is the HREF tag cause the SONY/ADE support to really get bad, more so than a single TOC entry for an HTML tag.

Try the attached fix on a calibre generated ePUB and you will see the difference.

=X=

[Ankh]

Originally Posted by =X=

Hi Ankh,
What my first post is getting at is the HREF tag cause the SONY/ADE support to really get bad, more so than a single TOC entry for an HTML tag.

I am talking about the same thing.

Unlike what calibre is doing, if your book is split at TOC entries (one XHTML file per TOC entry), there is no need for #HREF tags in TOC entries. All of them are referenced as per your fix.

[nrapallo]

Originally Posted by =X=

Okay here is my first pass

ePub_TOC_enhancer.pl eBookName.epub

=X=

As requested, Windows executable of your Perl script and batch file to reproduce same using the PAR-Packer-588 (v0.973) from CPAN. Now attached to post #1 above.

[astra]

Guys, you are geniuses!

[dordale]

I am looking forward to trying this when I get home tonight!

Thanks!!!

dordale

[doreenjoy]

I've been trying but can't get the script to work. Will have to try when I don't have a migraine.

[JSWolf]

I don't recall the error, but 0.01 did not work on an ePub I tried it on that had no DRM.