Remove the non-indexed elements?!

[un_pogaz]

When opening an ePub, Sigil only loads the indexed files into the <manifest>, and will delete any non-indexed files.
I understand this behavior which aims to remove parasitic files added by other unscrupulous and indelicate software.

However, more than once I have opened an ePub and I have only HTML pages and no images or CSS sheets.
However, its files are present in the ePub, simply they are not indexed in the <manifest> (probably because of unscrupulous software that confuses <manifest> and <spine>)

What I reproach Sigil for is the unilateral decision to delete non-indexed files, without warning or consultation of the user, because the risk of completely breaking an ePub is important.
It would be nice if, when opening, Sigil checked the correspondence <manifest>/ "content of the ZIP archive" and asked the user if he wanted to add them to the <manifest> in case of an non-indexed file as found.
The must would be that we are a check-list of files that we want to keep or not.

[DiapDealer]

Sigil's every action is predicated on the notion that the opf is entirely correct. And every action Sigil undertakes ensures that the opf stays that way. Sigil can't warn you about unmanifested xhtml files when it opens an epub, because it doesn't know they exist. It's not purposely deleting them. It hasn't made a "unilateral decision." It just hasn't loaded them. Because the opf (the boss) didn't tell it to do so when it was being parsed. What you're asking for would take a complete overhaul of how Sigil opens epubs, or a complete overhaul in how they're saved. Neither would be a trivial undertaking.

[KevinH]

Sounds like you should create an input plugin that walks the contents of the epubs zip file and adds any files present but not manifested that are css, image, font, or xhtml files to the manifest, and the xhtml files to the end of the spine in a some pattern (but what order?). You might want to include unmanifested javascript files as well.

It appears that these bad "epubs" are nothing more than a zip of a website that has been scraped.

Alternatively, unzip these "books" first and then use Sigil's Add Existing ... menu to pull in the pieces you want.

[KevinH]

Also since a spine is made up from manifest ids, any file not manifested can not be in the anyway. So no spine order would be relevant either.

That is not even close to being an epub. That is just a zip archive.

[DiapDealer]

Someone is clearly changing the extension of an .htmlz archive to .epub in my opinion. The metadata.opf file they often contain is woefully insufficient for the purposes of epub editing.

[DNSB]

Originally Posted by un_pogaz

What I reproach Sigil for is the unilateral decision to delete non-indexed files, without warning or consultation of the user, because the risk of completely breaking an ePub is important.

If the files in the archive don't match up to the files in content.opf, the epub is already completely broken so it's a bit late to talk about the risk of completely breaking an epub. Very rarely—if ever—does Garbage In produce Gospel Out.

A couple of quotes:

From the epub3 standard:
All Publication Resources must be referenced from the manifest, regardless of whether they are included in the EPUB Container or made available remotely.

and from Mukli Krisztián's epub boilerplate:

The content.opf file is the most important part of the EPUB package, because it defines the structure of the eBook and the metadata.

Manifest Section – This section is a list of all the content files, media, fonts, and stylesheets used in the eBook. The files can be listed in any order. However, you should not include a file in the Manifest Section that is not in the EPUB package. Also, you should not have undeclared files in the EPUB package that have not been declared in the Manifest Section.
[/QUOTE]

[un_pogaz]

I agree that, theoretically, an unindexed file in the <manifest> "does not exist" in the ePub, and that therefore technically the ePub is broken.
But not really either because the files are well used (in the <link> and <img>), but their non-importation breaks the ePub even more.
Problems that could have been fixed become unfixable.

Is a conversion problem, possible. But the idea is that not everyone is also respectful of the standard, so a little caution will be good.
The most vicious thing is that sometimes some files are indexed, others are not.

I am not asking for a systematic addition but just a check, which can be done using the "file table" of the ZIP archive. And no need to decompress any files, just load the OPF in memory and read the entries of the <manifest> (if Sigil was in C#, I could tried it).

Then Sigil works normally, and creates a conforme ePubs.

If it proves too complex to implement, okay, very sad, but I just want to reported this problem and ask for some thought to solve it.

To answer KevinH: So you ask me to 1) open with a "ZIP opener" each ePub I want to work 2) looked in the OPF the correspondence <manifest> contained in the ZIP 3) added the entries "forgotten" ? By hand?
This is possible, but the goal of a software is it to automate tedious, repetitive and sometimes complex tasks? Might as well take advantage of it.

[Thasaidon]

Have a look at the "Modify ePub" plugin for Calibre. This plugin can perform certain jobs on ePubs including "Add unmanifested file to Manifest".

If you run the "Modify ePub" plugin on your ePub before opening it in Sigil it may very prevent the problem you are having.

[un_pogaz]

Thank, I'd look at that.
(But integrating this security feature would always be a plus)

[DiapDealer]

What you're trying to open is not technically even an epub. It's just been given an .epub extension. The opf file in that kind of archive is for metadata purposes only. Sigil's not going to accommodate any-old zip archive full of html/css/images just because the extension claims it's an .epub. It must actually BE an epub (or it will be forced to be). And being an epub means certain strictures have to apply.

[KevinH]

And you can still unzip the epub and then use Sigil's Add Existing ... menu to bring in whatever files you want.

[AlanHK]

If I open a good epub in Sigil, delete the declaration of the image in the opf and save it, the image is still in the epub archive.

Then if I reopen it, there is no warning. The F7 check says "No problems found!" (even though there is still a page that references the image).

-- epubcheck says :

WARNING(OPF-003): Item 'OEBPS/Images/img001.jpg' exists in the EPUB, but is not declared in the OPF manifest.
ERROR(RSC-008): Referenced resource is not declared in the OPF manifest.

If I then make any random change (clean HTML, e.g.), save, the new file has lost the image.

This is perfectly correct, but a warning that something is amiss on opening would be helpful. You can unzip it and import the image or whatever; but you need to know the file was there.

So the workaround is to do an epubcheck immediately on opening a new file, then exit without saving if you see warning "OPF-003".

[DiapDealer]

Originally Posted by AlanHK

If I open a good epub in Sigil, delete the declaration of the image in the opf and save it, the image is still in the epub archive.

The question is: why would you do something like that? Anyone knowledgeable enough to risk manually editing the opf should already know that they also need to delete the image-file itself for the removal to be complete. Those NOT knowledgeable enough (or those who are just plain lazy like me), would use Sigil's integral delete function which would take care of both details properly.

Sorry, but protecting a user from every single way they could possibly manage to break their own epub if they try hard enough is not a rabbit-hole I plan to start down. Neither is protecting a user from all the ways an externally created epub could already BE broken before Sigil ever touches it.

Perhaps that will all change if we ever get to the point where sigil doesn't need to enforce its own internal structure on all epubs. but until that time ... it is what it is.

[Vroni]

Originally Posted by un_pogaz

Is a conversion problem, possible. But the idea is that not everyone is also respectful of the standard, so a little caution will be good.
The most vicious thing is that sometimes some files are indexed, others are not.

I am not asking for a systematic addition but just a check, which can be done using the "file table" of the ZIP archive. And no need to decompress any files, just load the OPF in memory and read the entries of the <manifest> (if Sigil was in C#, I could tried it).

Nice idea for an input plugin.

[Hitch]

Originally Posted by DiapDealer

The question is: why would you do something like that? Anyone knowledgeable enough to risk manually editing the opf should already know that they also need to delete the image-file itself for the removal to be complete. Those NOT knowledgeable enough (or those who are just plain lazy like me), would use Sigil's integral delete function which would take care of both details properly.

Sorry, but protecting a user from every single way they could possibly manage to break their own epub if they try hard enough is not a rabbit-hole I plan to start down. Neither is protecting a user from all the ways an externally created epub could already BE broken before Sigil ever touches it.

Perhaps that will all change if we ever get to the point where sigil doesn't need to enforce its own internal structure on all epubs. but until that time ... it is what it is.

Y'know, we get crap ePUBs like this, (web archives or the like) or I should say, inquiries about them, all the time, and IMHO, the probable best way to handle that is to follow Thasaidon's suggestion.

Run it through Calibre, manifest the unmanifested items, and then you can edit in Sigil, if you wish. Gets you everything you need, and it's a de minimis additional effort.

Given Sigil's intent, and what's trying to be done here, I don't see Kevin and Diap changing this anytime soon. Diap is right--protecting every single user from every single possible way that they can screw s**t up is the road to hell and bloat.

Hitch