Sigil 0.9.5 breakes some of my epubs

[Klecks]

Hi all,

I just stumbled on a problem using sigil on some of my epubs.
Please - can someone look into this and tell me if it's only me or a bug in sigil?

Sigil 0.9.5
Windows 10 (64)
I use both - sigil and calibre.

Usually I edited my books with sigil first and they got the sigil-wanted internal folder structure.
If I later used calibre on these books, calibre didn't care about sigil's wishes and sometimes inserted additional files and folders outside the sigil-given folders. But that's OK, because everything is linked properly and the books validate with epubcheck.

Problem: if I now open these books with sigil 0.9.5 again, sigil does it's usual I-sort-everything-my-way thing and rearranges the calibre-inserted files in it's own folder-structure; but it apparently misses to update the links! So I get broken epubs ...
I managed to produce an example-book. It's one of my problem-books (obfuscated for copyright-reasons).

By the way: on my older computer I use win 7 and sigil 0.7.2.
Sigil 0.7.2 doesn't have that problem - it updates all links. I tested my example-book with both sigil-versions:
open with sigil 0.7.2 - book OK.
open with sigil 0.9.5 - book broken.

For the moment I stick with using sigil 0.7.2 on these books - but maybe someone can look into that problem?

Thanks
Klecks.

[Notjohn]

Well, the obvious solution would be to keep Calibre out of the loop. I love Calibre for its intended purpose, manipulating my library of e-books, but long ago I was so horrified by the mess it made of my html that I resolved never to let it near my own books. If 0.9.5 otherwise works for you, surely that would be best. Why mix wine and beer?

[Klecks]

I can see that now, but harm is already done.
I have several hundreds of books like that in my calibre library.

[DiapDealer]

No need for a Sigil vs calibre throwdown. If everything is working correctly, there shouldn't really be anything one does that the other can't handle without "breaking" the epub. I'll take a look at this as soon as I can (if Kevin doesn't beat me to it ).

[DiapDealer]

Yeah, the test epub is a bit of a mess structurally, but it is valid, so Sigil should be able to handle it. But it doesn't. The files are being moved to Sigil's mandated structure OK, but the files' entries in the opf manifest whose hrefs start with the relative "../" are not being updated to the their new locations during Sigil's normal Source Updates. Will have to check into that.

As a workaround in the meantime, using the "Tools->Arrange into folders" feature of calibre's editor (on any epubs you edited with calibre) to make sure files are arranged into Text/, Styles, Images/, Fonts/ folders (OPF and NCX fields empty) will ensure that Sigil will be able to open the epub and successfully update the links. You must, of course, save the epub after making those changes in calibre's editor.

Thanks for pointing out the problem.

[KevinH]

Yes, having something be above the content.opf (in a higher directory) has probably never been tested. I have seen equal and of course in subdirectories but I don't think I have ever seen one where you go up a directory from the opf. That said, the way we build up the absolute url from the relative should properly handle this case unless the filenames don't match exactly in case.

I will try to look at this today if I get some free time.

KevinH

[DiapDealer]

As far as the OPF is concerned, part of the problem is that paths like "OEBPS/../cover.jpg" passed in the keylist parameter to xmlprocessor.performOPFSourceUpdates() are not matching the ../cover.jpg href in the OPF manifest ... so no updates are performed on those entries (even though the new/correct path is available in the valuelist parameter).

os.path.normpath-ing the keylist entries will allow the match to happen and the OPF file will be properly updated, but that does nothing for the NCX entries, or any of the links in the xhtml, or the xref:link parameter in the SVG cover code. I have a feeling there's a more universal change to be made that will handle the overall situation better that I'm missing.

[eschwartz]

Originally Posted by Notjohn

Well, the obvious solution would be to keep Calibre out of the loop. I love Calibre for its intended purpose, manipulating my library of e-books, but long ago I was so horrified by the mess it made of my html that I resolved never to let it near my own books. If 0.9.5 otherwise works for you, surely that would be best. Why mix wine and beer?

Mainly because you are spreading FUD?

calibre does not make a mess of your HTML unless you foolishly decide to use the conversion feature instead of the Editor.
Which is of course entirely appropriate for end-users, but you aren't an end-user so why on earth would you use the conversion feature?


I really don't understand why some people insistently continue to fraudulently compare the Sigil ebook editor to the calibre automated ebook converter.
Rather than comparing the Sigil ebook editor to the calibre ebook editor.

...

As DiapDealer said, it might be a good idea to use calibre's Arrange Into Folders tool -- which I specifically requested so people can manually set up a standardized folder structure like Sigil does automatically.

Also, this is quite obviously a Sigil bug (and not dependent on using calibre, even if calibre usage is likely to trigger the bug) -- so why bash calibre when Sigil will soon work perfectly and they will all play nice together anyway?

[Klecks]

Peace - I like both: Calibre and Sigil.
And apparently I managed - by using both - to create epubs, that are "unique".

Thanks for your help.
Klecks.

[KevinH]

Wow, that test.epub structure is a real mess. The epub spec in section 2.2 of the standard has only this to say about things outside of the META-INF:

"It is *recommended* that the contents of the EPUB Publication be stored within its own dedicated directory under the container's root."

Notice it says "recommended" not "required". This test.epub has some pieces both inside and outside the OEBPS folder (ie. jpegs outside of the folder) which typically is the "recommended common folder" to hold all of the publication pieces. But this is only a "recommendation" and not a "requirement", so Sigil should properly handle it.

So I will take a closer look at it.

Thanks for the head's up.

KevinH

[KevinH]

Now hopefully fixed in master.
Thank you for your bug report.

KevinH

[KevinH]

Hi DiapDealer,
IMO, the only sane way to do this is to build the original path from the ebook root (normalized) to the file's original location in the epub and use that as a lookup key linked to its new position inside Sigil's ebook structure.

Then for each file that has links that need updating, take its original position relative to root, add it to whatever href or url that is linked, then normalize it (QDir::cleanPath or similar functions) to generate a string to match against the the set of updates key value pairs generated when the epub was imported by walking the manifest.

This way works but we never ran into a case where the key itself was not properly normalized as most paths relative to the the ebook root were OEBPS/something without any "../" . So your solution of normalizing the key is correct and works as long as we normalize to ebook root, the value we look up to match how the key was designed.

So unless I am messed up, I think your approach is the same as mine, it just means we need to normpath both the key and the lookup value based on the original file location plus the decoded url as well.

The bug was we never ran into a case where the key itself had an ../ in the path (ie. other epubs followed the epub spec recommendation) but now we handle that and normalize the key itself given some manifest links now had "../" in them.

Take care,

KevinH

Originally Posted by DiapDealer

As far as the OPF is concerned, part of the problem is that paths like "OEBPS/../cover.jpg" passed in the keylist parameter to xmlprocessor.performOPFSourceUpdates() are not matching the ../cover.jpg href in the OPF manifest ... so no updates are performed on those entries (even though the new/correct path is available in the valuelist parameter).

os.path.normpath-ing the keylist entries will allow the match to happen and the OPF file will be properly updated, but that does nothing for the NCX entries, or any of the links in the xhtml, or the xref:link parameter in the SVG cover code. I have a feeling there's a more universal change to be made that will handle the overall situation better that I'm missing.

[DiapDealer]

Originally Posted by KevinH

So unless I am messed up, I think your approach is the same as mine, it just means we need to normpath both the key and the lookup value based on the original file location plus the decoded url as well.

The "same" except that yours seems to work for everything instead of just the OPF file (which is all the farther my experimentation/debugging had taken me).

I knew the problem probably needed to be addressed farther back along the trail, somewhere, I just hadn't located where best (and exactly how) to do it.

Thanks for the explanation. It (and your solution) made perfect sense.