Remove the non-indexed elements?!

[AlanHK]

Originally Posted by DiapDealer

The question is: why would you do something like that?

To make a test file.

Originally Posted by DiapDealer

Sorry, but protecting a user from every single way they could possibly manage to break their own epub

No one is asking you to stop people shooting themselves in the foot.

The issue is about how Sigil deals with epub files that have such problems already.

If the manifest does not match what is actually in the archive, then I think most people would like to know that, rather than having files silently deleted.
It could just give a warning that the named files will be deleted on saving. Or even give an option to add them to the manifest as if they had been imported normally.

[DiapDealer]

What you and some others aren't understanding is that it's not as simple as just "pop a warning when things don't match." In order to warn a user that an unmanifested file exists when opening/saving, Sigil would first have to KNOW there's an unmanifested file when opening/saving. And I can assure you that it does not. And MAKING sigil aware would require an overhaul of the open/save process that is simply not in the cards right now. Sorry. Patches welcome, though. I'll happily code the warning if someone provides the rest.

Sigil isn't deliberately deleting unmanifested files, it's simply failing to SAVE a file it doesn't know exists--couldn't possibly know exists without a double-parsing of the epub (opf manifest and internal archive file list) and file-by-file comparison of the two each and every time an epub is opened/saved. I'm not interested in tackling that.

[KevinH]

Again, since an epub is a zip file. Unzip the problem epub outside of Sigil. Then open a copy of the epub in Sigil. Navigate to Files->Add->Existing Files... and add any missing files (xhtml, images, fonts, styles) you feel are important but missing in the manifest by visually comparison to the unzipped version.

If you are afraid you might have a problem epub but are unsure, simply run epubcheck on it, which will happily report there are files that are unmanifested.

This can all be done right now with no changes to Sigil.

If this problem is so very common for you, then create a plugin that can parse the opf and compare it against the files provided in the epub/zip and do what you want.
This is not really that hard. If you do not know python, think of it as an excuse to learn or ask for help here.

[jackie_w]

I really hope I'm not going to offend anyone with this reply ...

Sometimes it's beneficial to have 2 tools which basically do the same job, i.e. Sigil and calibre Editor, because both have some features (and problems ) the other doesn't have. This might be a case in point.

If an image is present in an epub but is unmanifested, the calibre Editor will display it in the images file section marked as unmanifested.

I'm making no subjective statement about which is "the best epub editor" just that you can't go far wrong if you use each tool for what it does well

[DiapDealer]

Originally Posted by jackie_w

I really hope I'm not going to offend anyone with this reply ...

Sometimes it's beneficial to have 2 tools which basically do the same job, i.e. Sigil and calibre Editor, because both have some features (and problems ) the other doesn't have. This might be a case in point.

If an image is present in an epub but is unmanifested, the calibre Editor will display it in the images file section marked as unmanifested.

I'm making no subjective statement about which is "the best epub editor" just that you can't go far wrong if you use each tool for what it does well

No offense taken whatsoever.

[theducks]

Originally Posted by jackie_w

I really hope I'm not going to offend anyone with this reply ...

Sometimes it's beneficial to have 2 tools which basically do the same job, i.e. Sigil and calibre Editor, because both have some features (and problems ) the other doesn't have. This might be a case in point.

If an image is present in an epub but is unmanifested, the calibre Editor will display it in the images file section marked as unmanifested.

I'm making no subjective statement about which is "the best epub editor" just that you can't go far wrong if you use each tool for what it does well

Which is why my work flow uses both. I prefer the way different parts of each C and S operate.
As for problems with the structure of an EPUB: The Quality Check Plugin (for main Calibre) finds (not fixes) things like un-manifested files, unused images and many more. I tag em for remedial action using my fav tools

[un_pogaz]

The problem with KevinH's method is that it assumes that we know before opening the ePub that his <manifest> is broken.
This is the real problem.
Afterwards we can still trim the work with Quality Check, but once we have made this list, we must check ALL the library for each addition of books.
Personally, I use Sigil a lot to reorganize the content of the ePub (when I have to rework a lot of images inside, Open > Save > Ectract to Folder > Edits images > Return images in the ePub), so a false manipulation is quick to happen.

Originally Posted by DiapDealer

What you and some others aren't understanding is that it's not as simple as just "pop a warning when things don't match." In order to warn a user that an unmanifested file exists when opening/saving, Sigil would first have to KNOW there's an unmanifested file when opening/saving. And I can assure you that it does not. And MAKING sigil aware would require an overhaul of the open/save process that is simply not in the cards right now. Sorry. Patches welcome, though. I'll happily code the warning if someone provides the rest.

Sigil isn't deliberately deleting unmanifested files, it's simply failing to SAVE a file it doesn't know exists--couldn't possibly know exists without a double-parsing of the epub (opf manifest and internal archive file list) and file-by-file comparison of the two each and every time an epub is opened/saved. I'm not interested in tackling that.

DiapDealer, this new formulation of why Sigil remove the non-indexed elements is clearer than the first one given earlier.

You didn't want to take of this feature and I understand, it was marginal and there are probably other more urgent and useful features.
Especially if we have to review all the mechanisms for opening and extracting files... yeah, I understand that this is going to be tricky.

The problem is now known, added it in the bug tracker in "low priority/very difficult to solve" is all I asked for.

Have a good day.

[Hitch]

Originally Posted by un_pogaz

The problem with KevinH's method is that it assumes that we know before opening the ePub that his <manifest> is broken.
This is the real problem.
Afterwards we can still trim the work with Quality Check, but once we have made this list, we must check ALL the library for each addition of books.
Personally, I use Sigil a lot to reorganize the content of the ePub (when I have to rework a lot of images inside, Open > Save > Ectract to Folder > Edits images > Return images in the ePub), so a false manipulation is quick to happen.

I have to admit, in ten years, I've never--never--encountered this issue, FWIW.

DiapDealer, this new formulation of why Sigil remove the non-indexed elements is clearer than the first one given earlier.

I don't think that anyone here is lacking clarity on what you've requested, but it's still your request and your issue, created, apparently, by your workflow. Perhaps you should consider reviewing your workflow, if you frequently somehow end up with unmanifested items? How DOES that happen to you, anyway?

You didn't want to take of this feature and I understand, it was marginal and there are probably other more urgent and useful features.
Especially if we have to review all the mechanisms for opening and extracting files... yeah, I understand that this is going to be tricky.

The problem is now known, added it in the bug tracker in "low priority/very difficult to solve" is all I asked for.

Have a good day.

Yes, IF it were a bug, but it's not. It's how it's designed.

Look, if you run into this issue often, why not do what 3 or 4 of us have already said? Open the thing in Calibre, and manifest any mystery items. THEN put it into Sigil.

Or, as Kevin has mentioned--twice now--unzip the ePUB outside of Sigil. Open the file in Sigil, and ADD any items you need to. MANY of us work this way, to add things to an ePUB file that we need, for whatever reason.

I think that maybe, maybe, I've seen one other person here, since I joined something like 9 years ago, mention something about Sigil eating unmanifested files. Of course, my memory could just be wrong. If this happens to you often, or if your workflow creates unmanifested items, just use Calibre to get them manifested. It's the work of 2 minutes to do. It seems, quite bluntly, that you're asking Diap and Kev to put in a crapload of work, to create a function that, to the best of my memory, no one else has ever requested.

Given that this is volunteer-supported software, IMHO, they need to focus on those things that a) they know require time and effort, to maintain it and b) those things requested by the most people, (aside from the endless "make it like a word processor!" requests).

I'm sorry, and I don't mean to get sideways with you here, but afaik, it's NOT a bug. Putting it on the bug tracker implies it is, when it is not. Seriously, if this were happening to me often, I'd take a moment and look at my workflow, and figure out WHY it's happening, and address that.

Hitch

[un_pogaz]

I didn't create this ePubs, but found it as it is, from "various" sources (I am an archivist)
It was really very rare and it was especially me who was not lucky to have encountered such a problem.

As it is precisely rare, random, difficult to perceive and on existing files, it is for me unthinkable to have to check all the books.
The problem is not to fix the <manifest> but to know that it must be fixed (I agree that KevinH's method works).
What if I save the ePub without knowing that there were non-indexed files.... byebye, the files are losts.

I have launched a "Modify ePub" on my entire library, and I would launch a "Quality Check" tonight, but the risk will always be present for the new books that I will import and open in the stride.

I agree that given the design and operation of Sigil:
1) It is difficult to qualify his normal bug behaviour.
2) That you are a volunteer on the development of Sigil.
3) That there are other much more useful feature to implemented.
4) That apart from this very specific and rare problem, I have never encountered a bug (ah, once, 9.12: INI + UTF-8 )

For me, the subject is closed.
Thank you for answering my problem. Even if it won't be solved, at least I understand the why, and that some avenues of help were given, so I could act accordingly.

[Tex2002ans]

Originally Posted by Hitch

I think that maybe, maybe, I've seen one other person here, since I joined something like 9 years ago, mention something about Sigil eating unmanifested files. Of course, my memory could just be wrong. If this happens to you often, or if your workflow creates unmanifested items, just use Calibre to get them manifested. It's the work of 2 minutes to do. It seems, quite bluntly, that you're asking Diap and Kev to put in a crapload of work, to create a function that, to the best of my memory, no one else has ever requested.

I PMed a report to KevinH a few months back about an "EPUB file" with unmanifested 0 byte Icon files:

https://superuser.com/questions/2987...n-os-x-desktop
https://apple.stackexchange.com/ques...in-a-directory

Sigil refused to even open the "EPUB"s.

And epubcheck complained about it:

Spoiler:

WARNING(OPF-003): C:/Files/Sample.epub(-1,-1): Item 'Icon' exists in the EPUB, but is not declared in the OPF manifest.

These Icon files can happen on Mac OSX:

What is it?

It's name is actually Icon\r, with \r being the carriage return 0x0D. If letting the shell autocomplete the path in Terminal, it yields Icon^M, ^M being \r.

Icon^M is a file existing in all directories that have a custom icon in Finder. If you change a directory's icon e.g. in its Get Info dialog by pasting an image into the icon in the upper left corner, the Icon^M file is created.

Changing a volume's icon creates a hidden .VolumeIcon.icns file instead.

I had to:

1. Manually open the EPUB as a ZIP, and delete the Icon files. Sigil opened it fine.

OR

2. Open it in Calibre.

KevinH's explanation was approximately "even though these have the extension of EPUB, they aren't technically EPUBs", and he also answered this:

Originally Posted by KevinH

As it stands now, unless I see a bunch of these come from some manufacturer, I won't be changing Sigil to deal with such broken epubs. It would be better to use an unzip tool to unzip the epub removing all of the Icon files and then using the FolderIn plugin to actually load the files you want.

[...]

Since bad file names are a security issue in zips, I am quite happy the Sigil can not open this "epub". I would strongly recommend that the empty Icon files all be removed.

How the "EPUB" initially got created? I'm betting it was a poorly coded "EPUB creation" program that tried to just ZIP up entire directory structures and create the most basic of EPUB frameworks (content.opf, etc.).

[KevinH]

For the next release, I will look into at least generating a warning about an epub that has unmanifested files outside of the META-INF folder. I do not want to have Calibre BookMark files set off the warning.

How does that sound?

KevinH

[un_pogaz]

I have seen (with the update of the translation) that the warning of non-indexed files will soon be there.

I just have one small remark to make: I find the message a little too brief. Perhaps it would be necessary to add an additional paragraph on the possible consequences "Some images may not appear and may be deleted from the ePub"
But, detail.

[DiapDealer]

Originally Posted by un_pogaz

I just have one small remark to make: I find the message a little too brief.

YOU find it too brief? Or you're worried others will? Because I'm having a hard time believing that you, yourself, wouldn't be sufficiently warned that there's a problem with the epub you're trying to open. Especially after all the discussion here concerning unmanifested files in an epub.

The existing warning is enough. To those who heed warnings, it will more than suffice. To those who don't, no amount of blathering on about the inherent dangers of continuing will. Besides, using the word "images" in such a warning wouldn't be appropriate at all. ANY unmanifested file will be dropped from the epub, not just images. That message would be confusing to someone who got the warning when opening an epub that didn't have any images: "Oh well heck, I can safely ignore that warning. My epub doesn't have any images anyway!"

Besides ... what the new translatable string didn't tell you is that there's more to the message anyway. There's a detail button that will include the file names of all unmanifested files.

[stumped]

Originally Posted by un_pogaz

To answer KevinH: So you ask me to 1) open with a "ZIP opener" each ePub I want to work 2) looked in the OPF the correspondence <manifest> contained in the ZIP 3) added the entries "forgotten" ? By hand?
This is possible, but the goal of a software is it to automate tedious, repetitive ..

Seems to be a self infliced problem. Maybe you should stick to genuine epub from reputable sources, and stop with the dodgy sources that are giving you this zipped up and renamed crap. Then there would be no need for steps 1, 2, 3..

I have never seen a purchased epub with the issues you are describing

[AlanHK]

Originally Posted by stumped

Seems to be a self infliced problem. Maybe you should stick to genuine epub from reputable sources, and stop with the dodgy sources

Most of what I use Sigil for is fixing dodgy, imperfect files, usually conversions done in many and various and undocumented ways. The rest is converting my own files from DTP, a system that requires constant tweaking.