Problems with PML Conversions - Is it a bug?

[JulianL]

Since this is my first real post apart from my introduction (https://www.mobileread.com/forums/showthread.php?t=90175) I should start out by saying a big thank you to Kovid for Calibre - what a fantastic program.

I'm converting some books from my old Palm eReader days so that I can copy them into my iBooks library on my recently purchased iPhone. So far I'd say I've been 95% successful and have done quite a few conversions that look really good but I am having one generic issue that seems to be impacting many of my conversions. I wonder if it's a bug or whether someone can suggest a workaround.

My issue is with the \x and \Xn PML directives (http://www.ereader.com/ereader/help/dropbook/pml.htm). Calibre sees these as chapters to be added to the TOC but the problem is that it doesn't seem to differentiate the different levels so where I have something like:

\x\vPART I\v\x
\X1\vChapter One\v\X1

I would expect the TOC to look like:

PART I
   Chapter One

but what I actually get is

PART I
Chapter One

I tried changing \x to be explicit \X0 but I still got the same results.

I still have all the books that I'm converting loaded into eReader on my previous mobile phone so I can see what the original unconverted files are supposed to look like and the TOCs all show the correct indenting when viewing the originals.

Is this a Calibre bug? Is there anything I can do about it?

By the way, I am aware how far PML is down the list of prefered input formats that Calibre likes to work with so my expectations are suitably tempered but PML files are what I have to work with so I would like to try and get things as close to perfect as possible if I can.

- Julian

[kovidgoyal]

Sounds like a bug, you should open a ticket and attach a test case.

[JulianL]

Originally Posted by kovidgoyal

Sounds like a bug, you should open a ticket and attach a test case.

Wow. That was quick. Thanks Kovid. I just filed the bug #6194.

- Julian

[kovidgoyal]

Your bug may not be fixed as quickly though as the developer that wrote the PML input plugin is no longer very active.

[JulianL]

Originally Posted by kovidgoyal

Your bug may not be fixed as quickly though as the developer that wrote the PML input plugin is no longer very active.

Thanks. Sadly you've already been proved right I think. John just updated the bug with "Multi-level toc form PML input is not supported at this time." so I think I'm at the stage of trying to see if I can do some sort of workaround. Probably fewer than 20 of my books have multi-level TOCs and I don't mind investing a bit of effort to get it right if I know what to do.

As far as I can see from the manual and conversion options there is no concept of a TOC definition file to provide as input to the convertion process which was my initial thought.

Is it likely to be practical to run a normal conversion to generate a flat TOC and then go in and post-edit the ePub to fix up the TOC to the correct multi-level structure? I'm told that ePub is essentially just a zip container so if I download 7zip of something then should I be able to unzip and rezip the ePub to see and edit what's inside or are there better tools to use for this?

- Julian

[kovidgoyal]

Use Sigil to edit the TOC in the epub

[JulianL]

Originally Posted by kovidgoyal

Use Sigil to edit the TOC in the epub

I got some wierdness when I tried to use Sigil to edit the TOC on my Calibre-converted books.

As stated previously in this thread, the Calibre PML -> ePub conversion did generate a TOC that has all the expected entries in it when accessed from the built in Calibre book viewer and when downloaded to my iPhone and accessed via iBooks, the issue is that the TOC is all at a single level rather than nested. I was hoping that when I loaded the ePub into Sigil that I would see the flat TOC and could then set the necessary hierarchy levels but for some reason when I load an ePub that has one of these "seemingly OK but all one level" TOCs into Sigil then Sigil doesn't see any sort of TOC at all.

I really don't want to go through the actual text of the books to recreate TOC entries (mainly because I don't want to risk accidentally viewing some random but important pieces of text deep into the books as I process them and possibly spoiling a book for me when I subsequently some to read it).

My new plan is to post process my PML books using the xPml2XHtml python script that I originally got via links on the DarkReverser site. My searching here has already revealed that there is someone called KevinH on this forum who seems to be a developer of this script so hopefully he might still be around. I've already done a few experiments running the python script and then importing that xhtml book into Calibre and it seems to actually be slightly better in terms of the formatting of the text itself (there were a few little formatting issues that were tripping up the PML -> ePub Calibre conversion in a couple of isolated places).

As far as the TOC goes though the situation is one step forward and two steps back. I do seem to get the correct <h1>/<h2> nestings but for 2 out of the 3 books that I've tried I get a correctly nested TOC, with the correct number of entries, but every entry in the TOC is named "Untitled".

It's progress of a sort, and with this new approach I think I have a couple of things going for me that might mean I end up with a good result (i) The ability to very easily view the xhtml output by the python conversion and if I see anything that I think is tripping up the subsequent Calibre conversion then I might be able to remember enough of my Comp Sci degree to look at making changes to the Python script myself to patch things up, and (ii) If KevinH and possibly others are still around then this code path that I'm using might be more actively supported than the PML input converter.

I'll go and do some more experiments and post back if necessary, possibly under a new thread since I think we've determined what was going on with the X0 and X1 conversions before.

- Julian

[jackie_w]

I'm no technical expert and I'm only guessing...

Could you not take the converted epub with the single-level TOC and just manually edit the toc.ncx file in a text editor then reinsert the .ncx back into the epub?

[kovidgoyal]

Doesn't Sigil allow you to edit the NCX file directly, otherwise as jackie_w suggested, use a text editor. It's a simple, XML file that should be relatively straightforward to edit.

Yet another alternative is to use the TOC options to detect and create an multi level toc in calibre automatically.

[user_none]

Originally Posted by kovidgoyal

Doesn't Sigil allow you to edit the NCX file directly

Yes and no. Sigil ignores the NCX completely. It generates its own TOC based on h tags which you can edit.

[JulianL]

Jackie (& Kovid) - I've installed 7-zip and have used it to look inside one of the converted ePubs and have the toc.ncx file open in a text editor at the moment. Fixing this up in an editor is my fallback position. Currently for a given book I can get Calibre to either generate a TOC with all the correct entries and titles but no nesting (if I convert directly from PML), or a correctly nested TOC but will all the entries marked as "Untitled" if I convert from xhtml. I'm going to run both convertions, look at the format of the two TOCs, each broken in a different way, and I'm sure that by comparing the two I can work out what I need to do to fix it all up.

Kovid - If you mean using XPath expressions in the "Level 1 TOC ..." and "Level 2 TOC ..." in the TOC section of the Calibre convertion options rather than relying on the general "Detect chapters at ..." XPath expression in the "Structure Detection" section of the options then that's encouraging because that was definitely on my list of things to try. It'll be a nicer solution if I can get it right so that's what I'll try first.

user_none - That explains my experiences with Sigil. I'm glad in a way, I already have the 2 different options above to explore so I'll set aside Sigil as an option and try to get a satisfactory result with one of the approaches above.

Thanks everyone for all the various bits of advice. I'm pretty confident that I'll get there in the end, and hopefully without coming back here with too many stupid questions, but it's great to find everyone here so helpful.

- Julian