New message when I open Sigil 0.9.17

[dynabook]

When I open a file in Sigil I get a message that "Files exist in manifest which are not in epub. They will be ignored." What am I to do with this information? How can I find out what these files are? Are they unnecessary? Should i Find them and remove them???

--MH

[DiapDealer]

What OS? And the answer to "how you can find what files these are" is to click the details button on that dialog. It will list all the files.

First find out if the epub passes EPubcheck validation. Report back with that info and we'll go from there.

[dynabook]

I am running MacOS 10.14.6 (Mojave). The epub passes the epubcheck 4.2.2.

The details button gves a list of files. My question is: Does this list include the files not in the manifest? should I compare to the manifest?

--MH

[DiapDealer]

Yes. The list of files in the dialog box is the list of files Sigil _thinks_ is not in the manifest. That may or may not actually be the case. Does the list appear to be every file in the epub? And does this happen with every epub you open or just some?

Kevin's been looking into a similar issue reported by a Mac user, so I'll let him take over. He understands the specifics better than I do.

[KevinH]

We have a Mac debug build that will list the files it finds in the zip along with what it thinks in the manifest.

Please see this bug report:

https://github.com/Sigil-Ebook/Sigil/issues/448

It has a link to a special build for Mac to debug just this type of error. It has instructions on how to download, unpack, and run it with a special environment variable.

If you are willing to help us debug what is going on please check out this Sigil issue which has all the info you should need to help track this one down.

I think that build also has some changes to how files are launched from macOS with open with and drag or drop, so you could report on whether they help or not.

Please let me know if you are willing to help out.

Thanks,

KevinH

Originally Posted by dynabook

I am running MacOS 10.14.6 (Mojave). The epub passes the epubcheck 4.2.2.

The details button gves a list of files. My question is: Does this list include the files not in the manifest? should I compare to the manifest?

--MH

[dynabook]

I read the bug report. I probably have done the following:

1. When Sigil was downloaded and installed last you did not manually drag it from your Downloads folder to your Desktop before mounting the dmg file and copying Sigil.app.
When this happens macOS's gatekeeper runs the app in a sandbox actually moved away from the applications location. This would cause all kinds of manifest warnings.

I will redownload and reinstall to see if that fixes it. Of course that will not test your other fixes but I will report back.

--MH

[dynabook]

Steps:

1. Remove Sigil and PageEdit from /Applications and empty trash.
2. Redownload Sigil and PageEdit from Github.
3. Pull Sigil and PageEdit DMGs from Download folder to Desktop.
4. Open DMG and drag Sigil to /Applications. Do same for PageEdit.

Run Sigil and receive same message as before. Get message for any epub opened. All files listed seem to be in manifest.

I will investigate using the special build.

--MH

[KevinH]

If you go farther down that bug report you will see the link to the special debug build of Sigil for mac and instructions on how to install and use it.

The debug log file created in your Desktop as a result of the test with that epub should tell us all we need to know.

Please give it a try and let us know what the debug log says (via pm here is necessary).

[dynabook]

Uploaded debug file after having to "man zip" and create Github account! Been a while.

Do you want me to post here as well?
--MH

[KevinH]

No need. I just grabbed it form the issue.

[KevinH]

This is so strange...

It walked the manifest file and built up the file paths from the temporary Directory that Sigil created to store the unzipped epub:

Here is the temporary directory used to unpack the epub into:

m_ExtractedFolderPath: "/var/folders/kl/5s_sbwr972n1qq9xrk7rlyvr0000gn/T/Sigil-DTjJwa"


Here is the complete file path used to store one of the files as an example:

Debug: creating manifest file path from: "/var/folders/kl/5s_sbwr972n1qq9xrk7rlyvr0000gn/T/Sigil-DTjJwa/OEBPS/Text/ded01.xhtml"


But when you ask for the epub root relative file path for that file it comes back with
something that makes no sense:

Debug: Manifest File Paths:
Debug: "-DTjJwa/OEBPS/Text/ded01.xhtml

instead of the expected:

OEBPS/Text/ded01.xhtml

It is almost as if the temp folder name has embedded null characters of some sort!

Not sure why this is happening, but there is nothing wrong with the epub itself. The problem is in how mac OSX 10.14 is generating temp folder paths. It seems to be embedding nulls or other invisible characters so that a count of the strings can not be used to remove the root folder from the name.

This is very strange.

I will have to dig deeper.

KevinH

[KevinH]

Okay this is because macOS is using symlinks in creating the tempfolder path that are not resolved:

dynabook/

Please do the following in Terminal.app
cd /
ls -al /var
ls -al /tmp

and report back what they say:

On my machine:

ls -al /var reports:
/var -> private/var

and ls -al /tmp reports:

/tmp -> private/tmp

[DiapDealer]

In this commit: https://github.com/Sigil-Ebook/Sigil...517017fbc659d5

Is m_ExtractedFolderPath canonical? And if not, shouldn't it be? If it's absolute, then you might be subtracting apples from oranges to get the relative path. Canonical should remove symbolic links.

[KevinH]

I think the problem is the call to create a canonical path for each manifest file. On my machine it will not resolve the symlinks, and on your machine it resolves the symlinks so it converts:

converts:

"/var/folders/kl/5s_sbwr972n1qq9xrk7rlyvr0000gn/T/Sigil-DTjJwa/OEBPS/Text/ded01.xhtml"

to

/private/var/folders/kl/5s_sbwr972n1qq9xrk7rlyvr0000gn/T/Sigil-DTjJwa/OEBPS/Text/ded01.xhtml

And when I remove the length of the m_ExtractedFolderPath which was:

"/var/folders/kl/5s_sbwr972n1qq9xrk7rlyvr0000gn/T/Sigil-DTjJwa"

I end up with leaving 8 characters too many in the path (length of "/private").

So this appears to be caused by a change in how canonicalFilePaths are being resolved in macOS 10.13 and earlier and macOS10.14 and later.

This would cause lots of havoc in just about everything to do with launching a new file with open with as well.

"canonicalFilePaths" are supposed to have all symlinks resolved but that does not happen on my machine but does seem to happen on your machine.

So I think this is a bug fix made by Apple for macOS 10.14 that breaks Sigil.

I will look into fixing this.

[KevinH]

Yes but using a canonical file for the temp folder created another issue for macOS a ways back (and if our call to canonical is broken - that explains why.

So the best solution is not to resolve relative hrefs from the manifest using canonical but instead to do it the way I have just added to PageEdit (a utility routine) that will work for all platforms.

What we need is an absolute path without relative segments if canonical is broken on macOS and sometimes doesn't resolve all symlinks that start at root.

I guess we could start with a canonical name in TempFolder and if that breaks me (my macos 10.13 machine) we can try to just resolve relative links and not symlinks and let the symlinks live. The problem is hard links will then break the canonical file paths as will some aliases. Hmm....

Kevin

Originally Posted by DiapDealer

In this commit: https://github.com/Sigil-Ebook/Sigil...517017fbc659d5

Is m_ExtractedFolderPath canonical? And if not, shouldn't it be? If it's absolute, then you might be subtracting apples from oranges to get the relative path. Canonical should remove symbolic links.