Sigil-1.5.0 Beta Released
 

Sigil-1.5.0 Beta

Sigil-1.5.0 Beta represents a mix of bug fixes and new features for both epub2 and epub3 users.
Due to large internal changes in the Sigil codebase, this release should be considered Beta (non-production) ready.

This build of Sigil has passed all of our primary tests and appears to be stable but the feature set of Sigil has grown so large that no single person uses it the same way nor exercises all of the code.

So we need your help to track down any remaining issues, especially related to the new features and workings made to Code View, the Metadata Editor, and our new CSS parser being used to identify unused CSS Selectors.


New Features:

• Can now handle single xhtml file sizes over 2megabytes in size via its own URL Schemehandler
• Will now highlight matched open close tag pairs while editing in Code View
• Delete open close tag pairs (Remove Tag Pair)
• Default selection of text for basic CodeView formatting including bold, italic, etc based on cursor position
• Double-click (and shift double-click) on a tag to select tag contents (including tag)
• Expanded split on Sigil Split Marker capabilities to work better with nested tags
• Added a new C++ CSS Parser and Query engine that works with Sigil's version of Gumbo
• The Reports tool for "CSS Selectors" now lists all CSS selectors not just classes
• The "Delete Unused Styles" tool now handles all unused CSS Selectors not just classes
• The Reports for "CSS Selectors" and "Delete Unused Selectors" now handle selectors in XHTML Style tags
• Added ability to load text and csv files to Group Saved Searches to automate lists of replacements
• Reworked the Metadata Editor to be much more Human Readable with tooltips to show xml tags
• Added semantic code to the descriptive field in Add Semantics as a learning aid (Thank You BeckyEbook!)
• Checkpoint ManageRepos now has the ability to sort the repo table by any column (Thank You BeckyEbook!)
• The Sigil User Guide has be completely reworked to bring it to Sigil 1.5.0+ levels

Bug Fixes:

• Fix issue with custom ncx names in non-standard empty epub layouts
• Fix Import Text to properly add ncx if missing for epub2
• Fix issue with "Delete Unused Styles" not properly detecting all used selectors
• Fix issues with repeated use of Mend and Prettify on bare text in structural tags
• Fix extra line issue with Link to Stylesheet (Thank you BeckyEBook!)
• Fix bug in id assignment in EPUB3 Metadata editor
• Fix bug in trailing slash in Move To Folder paths
• Fix bug in spelling of Columbia->Colombia in Languages (Thank you Tex2002ans!)
• Fix bug in Clip Editor pasting of multiple clips
• Fix bug in PerformCSSUpdates related to quoted string in content: values
• Fix bug related to iframe handling when loading Preview
• Fix typos in XMLEntities descriptions (Thank you BeckEbook!)
• Fix numeric table alignment to align right in multiple tables (Thank you BeckyEbook!)
• Fix bug in Add Existing not properly using QProgressDialog for long import
• Removed long deprecated and now invalid use of "altlang" in EPUB3 Metadata Editor

Please check the Sigil Wiki for important Sigil support links, additional resource downloads, and platform-specific trouble-shooting tips/requirements.

Mac users should still download and install ActiveState's ActiveTcl Community Edition to utilize plugins that use Tk/Tcl GUIs. More here.
Mac users should also check out the wiki entry on the New Release File Format

The latest Sigil user guide can always be downloaded from its own repository.

PGP Fingerprint

The binary downloads (and source) can be found as assets at the bottom of The Sigil Github Release page.

Post reserved for future use.

Exciting! Thanks! :)

Nice.

If it is ok to request something that I hope you will consider and may include, either with the upcoming version, or later, I have found the new error message upon EPUB open with possible errors a bit confusing.

There are times when I edit an epub outside of Sigil and may introduce syntax errors that may possibly lead to data loss as mentioned by Sigil. Then there are more frequent times when I open a commercial EPUB in Sigil and the possible fixes are perhaps only metadata such as DOCTYPE, missing head elements, etc. Generally if I am making an EPUB myself, I I don't mind running epubcheck before the next opening with Sigil in case I messed up something. In the case of editing commercial epubs, it'd be nice to not to have run epubcheck each time before first open with Sigil to verify it is ok. More detail of the errors would be nice and I think help avoid potential data loss and save effort of verifying the EPUB when it is unneeded.

Windows 10 and/or Edge is trying very hard to protect me from installing this version!

Normal at the start of each version. Retry now, they should be less annoying (10-20 minute and it's good).


Oh damn, the highlighting of the tags is so nice and will change my life.
Thanks you

Bug repport:
1) Display the characters escapes directly in the ToC editor (%20, %5B, %5D...)

2) Files with "&" in their name are excluded and deleted

3) ePub2 the property "Scheme" can not be customized (fixed and defined list) [image 1].

4) ePub2 impossible to change the value of a name attribute for the custom elements

5) ePub3 adding the metadata "Belongs to a Collection" should preemptively put the property "Type of collection" and "Position in the group" to facilitate editing (and preferably 2 metadata: one for the series and one for the collection/set)

6) ePub3 the very long custom element (several lines like those created by Calibre) are unreadable and impossible to modify [image 2]

7) impossible to select tag name, a attribute or the value of an attribute with double-click (over-ride by "select tag content" behavior)
I propose to reduce the behavior only when you Maj-Double-Click

Probably other more discreet features that I rarely use... well, it's a beta, so it's almost normal.

Thanks for the bug reports, but I am not sure what you exactly mean with them. Would you please add information explaining each one in detail?

It would help to know what is correct or expected against what you are seeing.

As for & in filename, unless it is % escaped in your opf it is will be unmanifested and therefore will not be loaded. That sounds like your manifest in the opf is incorrect on reading in the epub into Sigil. If so, you should probably file a bug report with whatever created this epub in the first place.

Also, using special characters in a filename is probably not a good idea for general acceptability.

For your 6. that is json, not css or xhtml. Nor is it xml. Exactly what is a "custom element" and if this is metadata, then does it even belong inside the opf?

For your 7, just drag to highlight. Shift Double-click has already been defined. That is the trade off made when that feature was added. (Is that what you meant by Maj Double Click?).

I was able to reproduce this issue with a file that was originally created with Calibre. If you add an ampersand to a file name in Calibre, it'll escape it as & instead of %26.

Apparently, the ampersand and the semicolon in & are escaped by Sigil and the file isn't found:

AFAIK, Calibre will sometimes add very long JSON-based metadata entries. I couldn't find an example, because I usually remove these entries. Hopefully, un_pogaz will provide an example.

Well the Sigil Metadata Editor will never support editing custom metadata full of json in the editor itself. For specialized things like that people will need to directly edit the opf and be careful to properly escape things themselves.

For your 5., that is an enhancement request not a bug, but it is a good idea and I can add something along those lines for Sigil 1.6.

is Image #2 an example of an "unreadable", JSON entry? Seems pretty readable to me.

But at far as editing goes: at some point, those who wish to use custom schemes, and embed formatted JSON data in metadata values might have to get their hands dirty and edit the opf file manually. Sigil's not going to provide a gui feature to be able to easily edit calibre's proprietary json format used to support custom columns, categories, and collections in calibre's library view.

I simply can not believe calibre's quality check approves of using an "&" in a file name inside an epub!

Even so urls/uris are supposed to be % encoded not xml escaped. So unless Kovid can point to the epub spec where "&" chars are supposed to xml escaped in urls or uri's in the opf, I consider this a bug in calibre not Sigil.

An & is not a legal char in a uri/url filename or path and you can not xml escape an & without using another &!

percent encoding of spaces, &, #, etc is how the spec says to deal with these chars.

That said, I could try to xml unquote the filename before % decoding but I think that will probably open a kettle of fish for something that is simply not a good idea at all.

Sorry but I am going to need to see where in the opf spec it says you can do that. It should have been % encoded and then no xml quoting or unquoting would be needed.

To workaround this, unzip the epub and use AddExisting to add the missing xhtml files. Sigil should properly % encode these when adding them to the opf.

For your 3., agreed but unrecognized dc:identifier schemes are not going to be supported in the MetaData Editor. Especially since for EPUB3, the scheme is no longer recommended and instead a urn: based prefix is added to the identifier.

For custom things like that, you will have to edit in the opf being careful to properly add any needed prefix definitions if needed.

Edit: but if there is a popular dc:identifier scheme that is not already in the list of recognized lists, I could add it to the list easily.

Edit2:
Looking at that image you posted, that dc:identifier is a uuid. It has nothing in particular to do with Calibre per se. Any software can generate a random uuid identifier. The scheme here should be UUID.

For your 1., I still do not understand what the issue is here? What exactly are you asking to be fixed or changed? A specific example would help clear things up.

Thanks,

Kevin

For speed reasons, Sigil will simply look for no errors when loading a document. Sigil's importer is not a full validator nor does it have the interface to be one. If any errors are detected, the user is warned that errors exist.

You do not need to worry about loading an epub that is well formed in Sigil. There is no need to pass epubcheck before being loaded in Sigil.

Sigil will warn only when it finds something it does not consider spec and will offer to automatically fix it for you.

I think you see that message about the possibility of data loss, and it gives you pause. Any fears here are very overblown. Actual data loss is very very rare and only happens due to the xhtml text not being at all parseable (ie so broken so that gumbo's parser can not read it).

Simple things like missing doctype, missing html tag, etc, will always be happily and safely be fixed by Sigil's gumbo parser during Mend.

In earlier versions of Sigil before Sigil 1.0, Sigil just quietly fixed these errors for you while moving all files to their Sigil standard locations.

From Sigil-1.0 to Sigil-1.3, once moving to "standard locations" stopped, errors due to missing doctypes started to happen but not be fixed or detected until another Sigil tool was run that needed gumbo.

From Sigil 1.4 onwards, we now properly detect the missing doctype and warn the user on load so that they run Mend (after load) or we do it for the them depending on user Preferences.

If you truly do not want to auto run Mend on open, set it to that in your Preferences. If you want to see what is broken that mend fixes, then checkpoint it after load and then run mend, and then compare it against the checkpoint.

I just re-tested this and Calibre Editor correctly escaped & as %26. (I don't know how I ended up with &.)

That is strange. So Calibre wants to % encode the & in the filenames in manifest href urls too. That is the spec.

So perhaps a conversion or plugin in calibre from another format generates these broken manifest entries?

Either way, this is not a Sigil bug. If desired the user can unzip the broken epub and use Add Existing to add in the xhtml files, images and css to properly create an epub with a working opf.

But the right way to maximize compatibility with epub readers it to not use reserved chars inside filenames in the first place.

As far as I'm concerned, Sigil is handling filenames with ampersands correctly. You can rename files within the epub to contain ampersands and Sigil will properly encode the path in the opf manifest. Sigil will also open any epub that has a filename with an ampersand so long as the ampersand is encoded correctly in the opf manifest (and the path exists) to begin with.

The only time there's a problem is if an existing epub has an opf with an invalid ampersand character in it. And even then, it's not because of the illegal ampersand character (directly) that files might be discarded when subsequently saving the epub. It's the fact that the illegal character makes the path to the file incorrect. Which means it's not properly manifested. And improperly manifested files have always been discarded when opening them. It's no different than a mispelling or a case sensitivity issue.

I am in 100% agreement with you. The only question in my mind is if we could make the ImportEpub.cpp code be more robust to broken manifest entries somehow, without adding potential problems.

Something like take each manifest entry and try the following:

1. try url % decoding manifest entry to see if it matches a file in the epub zip archive

2. if not, try xml decoding the manifest entry to see if it matches a file in the epub archive but report a url encoding warning in the manifest.

3. finally, if not, try taking it at face value (no xml decoding or % decoding) and see if the manifest href matches a file in the epub zip archive.

4. if not report the manifest entry url not being present in the epub as a warning...


After doing this for every manifest entry, report any leftover files in the epub zip as unmanifested.

That may help Sigil read in epubs with broken manifest hrefs.

I'd be up for anything that made ImportEpub less allergic to broken manifest entries. But are you talking about potentially fixing broken manifest entries, or just giving more meaningful warnings regarding them? The former sounds great, but I would think it could lead to other problems. The latter would still be an improvement and sounds less ... dangerous?

Hopefully fixing them upon import would be the goal. But you are right, this might cause issues and so would be something for Sigil-1.6.0 not Sigil-1.5.x.

This is a question asked from technical ignorance and I ask as sometimes asking such questions has lead to coming up with an elegant novel solution.

Rather than trying to fix the manifest would it be easier/cause less problems to generate a new one?

Essentially we do that on import but use the existing opf to guide that. The opf is supposed to have a valid manifest and files not listed in the manifest should be ignored. The opf should also have a spine indicating the order of the xhtml files. That spine order uses the manifest ids assigned to each valid file. So having a valid opf is important. No epub should ever be without one.

There are also file naming/path issues that can cause security concerns for maliciously crafted epub/zip archives.

So the best way to handle this is for the original epub to have a proper valid opf, manifest and spine included.

If not, you can still open this epub in Sigil by unzipping it and using Add Existing to add the xhtml files and Sigil will properly create a manifest. You would have to determine the correct spine order from a toc or from other sources.

So just ignoring the manifest and/or the opf is typically not a good idea. It can be worked around if broken and that is what we are discussing.

What I am uneasy about is I'm not sure what Sigil does in the cases when there are errors to fix, e.g., I may miss a close tag, or change some paragraph to header but make a typo such as <h2>…</p>. Of various errors, I am not sure if the automatic fix is what I'd prefer.

If it were possible to distinguish between such malformed errors and others such as merely fixing doctype or other metadata, that would save a lot of time rather than having to as you suggest checkpoint and compare. Often I am editing outside of Sigil as another editor I find quicker to load up an EPUB, and perform regular edits. I understand Sigil does not fully validate the EPUB; it would just be nice to have more clarity in the message.

Thank you for explaining. I bow to your superior knowledge.:thanks:

A lot of things, I'll answer as best I can to your request.

Crap, I just realized my interpretation/reading mistake for the 2) : False alert, sorry :smack:

The ePub "& amp" is the one with the 1) element
I want to specify that this ePub was perfectly valid in the 1.4, Sigil authorized me perfectly this character. The 1.5 failed to load this.

The ePub "JSON" contains the element 6) [copyright safe]
Video of the result in the editor https://youtu.be/YisMuSQRtHA

For 3) Yes, this is theoretically a UUID, so it's not the best example, but if I wanted to add a custom one (like "internal-editor-version"), I think having to go directly through OPF is brutal.
Maybe add at the end of the list a "custom identifier" entry that when selected allows direct editing of the scheme (possibly through a small dialog window)
(and I speak only about epub2)

The 4) is a bit the same problem [image], you can't edit the "header" of the Custom element.
Maybe it's my little comfort, but it would be convenient to not have to think « this I can edit here, this not I have to close the window open a file and find the only thing I'm looking for among 40 lines of raw text ».
(also only about epub2)

And sorry I wasn't clear in my first post.

In my tests, the same 10 files with illegal &amp; entities in the href attributes of the opf manifest give the exact same warning using both Sigil 1.4 and 1.5. Both versions (1.4.and 1.5) also discard the same 10 improperly manifested files in my testing.

Sigil uses Google's gumbo parser which follows the rules of html5 parsing standard according to the whatwg browser standard. In other words, it does exactly the same thing that Chrome, Edge, Safari, and Firefox, does when faced with that same broken code.

As for more details, once loaded Preview will show you exactly where and what well-formed errors exist on a file by file basis if desired.

Just turn off Mend on open. Load the epub "as-is". Hit checkpoint (one mouse click). Run Mend. Compare against the fixed code to see everything that gumbo fixed or changed.

Or instead of Mend, just run a basic well-formed check (see Sigil's menus), which is a basic validator that will show you just serious well-formed errors.

You have options. Loading an epub is not going to give you a complete list of errors like a validator will for speed and missing interface reasons but can load them "as-is" for you to validate, or hand fix, or can auto fix them for you.

Sorry. I missed your retraction of #2. I'll look at #1 with regard to the provided & amp.epub

Ah, okay.
I swear it works without a problem on mine 1.4.
The first version of this ePub dates back to at least 0.8, it's super old, maybe something survived from that period...

Anyway, it's clearly out of scope of Sigil. I'll fix it to be comply with the standard.

PS: For info, the bugged entries are referred as such:
Bug closed

:smack::smack::smack:
What an idiot I am, it is the 1 (the escape character) from which I retract, the 2 is "always" valid
:smack::smack::smack:

For your 4., yes, that is a bug. Custom elements need to be able to be specified once but making names editable in general is not a good idea as it would allow novice users to mess up things completely by changing names they should not be changing. That is why we locked down the names to begin with.

I will look into adding a small dialog prompt to query for a Custom metadata element (name) before adding it in table as once there, editing the name will not be possible but editing the value will of-course be.

Thanks,

Again... using your test epub, I can see no difference in how Sigil 1.4's ToC editor deals with the ampersands than Sigil 1.5.0 does. Both display the % encoded file names, and I can enter the ampersand character in either version and it is properly % encoded in the ncx (as well as the toc editor next time I open it).

You're going to need to be more specific about what you're experiencing with regard to ampersands that's different from the Sigil 1.4 series compared to the 1.5 beta. Step by step instructions of what you're doing and how version of Sigil differs when performing those steps. "Failed to load this" is sufficient. I see nothing that's failing to load in either version with regard to ampersands and Sigil's ToC editor.

I don't doubt there's SOME difference you're experiencing between the two versions, but I need much more in the way of particulars if I'm going to be able to recreate it.

@DiapDealer
No, no it's my fault.

I invert the number of the problem 1 and 2
The character escapement is done correctly in the ToC.

And after all, the 2 "problems" as solved.

OK, thanks. Just wanted to be sure.

I have skipped packaging this for now... I assume there will be a 1.5.1 soon that I should package instead?

That's the plan, yes. If no huge, deal-breaking bugs rear their ugly heads, 1.5.1 will be out before long. Currently, 1.5.0 isn't even triggering the built-in "update available" notice.

So to sum up ...

1. and 2.- Not a bug as & is not allowed in a url such as manifest href. Think about making the ImportEPUB more robust to mistakes like these for Sigil-1.6.X.

3. You will need to hand edit the opf to add truly custom identifier schemes in epub2. Such dc:identifier schemes are no longer used in epub3.

4. This is a bug. It is now fixed in my local tree that will prompt to input your custom element name before adding it into the MetaEditor. This fix could appear in a Sigil-1.5.1 but also since so rarely used, saved until Sigil-1.6.x.

5. This is really an enhancement request and not a bug - but one we should be able to squeeze into a future Sigil 1.6.x release

6. Editing huge blocks of unneeded JSON in Metadata elements will not be supported by the Metadata Editor. It will just treat it like any other text field.. Most of that info is unrelated to the book itself and is more saving state within the book. You will need to hand edit the opf if this is what you are looking for.

7. Yes, the ability to double-click on an attribute name or value inside a tag to select it is overridden by the select tag/ select tag contents ability we just added. The only way to deal with this is to reassign the newly added Double-Click in a tag to select its contents, and Shift Double-Click in a tag to select the tag itself and all its contents to some other modifier along with Double-Click. To select a name or an attribute inside a tag, you will have to click and drag.
Double clicking on a word outside of a tag still works just fine.

We could consider changing that for Sigil-1.6.x but we must then get some agreement from the people who wanted the select tag and select tag contents to either give up one of those two new selection modes up or decide on a new modifier for one. Straight double-clicking for selecting words can then be restored (anyplace).

Thanks for your bug reports! And thanks for testing our beta release!

My preference would be for consistency as I keep forgetting that double-clicking to select words doesn't work inside tags (whereas it's pretty universal everywhere else, not just in Sigil but other programs as well).

If people could be persuaded to use a new, different modifier for selecting tag contents, I would be in favor and much appreciative.

Thanks everyone for taking the time to respond to my feedback somewhat... raw.
All as been solved...
But in the continuity of Odamizu, I insist on the over-ride of the double-click.
Personally, it breaks my editing habits and is a gap from the conventions of selection behavior in other software.

I suggest the use of the following shortcuts:
Maj+Double-click for select the tag content
Maj+Alt+Double-click for select the all tag