Some problems/bugs in Sigil

[man2010]

These problems are related to conversion of Sigil-generated EPUB files to Kindle or adaptation of HTML code from a Kindle ebook to EPUB via Sigil.

Sigil usually converts the code of Unicode symbols into symbols themselves in UTF-8 files (when you import an HTML file into Sigil); later, when you try to convert such an EPUB file to Kindle, these symbols may get mangled.

Let's say your HTML file contains this line of code:

⌘ Hello →

After you edit this file in Notepad++ (some other lines of code) and save the file, this line of code will stay the same:

⌘ Hello →

In contrast, if you import this HTML file into Sigil, and then save the EPUB file, this line of code will get converted to:

⌘ Hello →

Thus, Sigil created a possible problem with encoding. There is no way to turn off automatic conversion of HTML entities into symbols in Sigil, except to specify individual HTML entities (inconvenient). Some EPUB readers may support HTML entities but not the resulting symbols. For instance, Internet Explorer (not an EPUB reader) reads HTML entity ’ but cannot read the corresponding symbol in UTF-8 files. There used to be a built-in validation tool in KindleGen 2.3 and 2.4, which produced warnings for various symbols; these warnings disappeared after you replaced the symbols with HTML entities or Unicode codes. This observation leads me to believe that the Kindle platform prefers HTML entities to symbols.

Another example, on another forum, a poster reported that musical score software Finale 2012 produces an EPUB3 file that converts OK to Kindle, but if you import this EPUB file into Sigil, edit, and save it, then the resulting EPUB file cannot be converted to Kindle (most musical symbols get corrupted). This problem existed 3 years ago. During recent testing, I was unable to find examples of such symbols (I don't have Finale 2012). It is possible that this bug was fixed in the latest version of KindleGen. Nevertheless, I found some strange behavior of Sigil with respect to Unicode symbols. If you test this list of symbols:

⌚ ⏛ ⏚ ⏰ ♕ ⛄ ⏳ ☻ ☼ ♪ ⏎ ⏏ ⊄ ⊆ ⊥ ∏ ∉ ⓳

In Sigil, all these pieces of code get automatically (and successfully) converted into corresponding Unicode symbols, except Sigil cannot show many of them (you see a white square in HTML and book views). When you convert the resulting EPUB into Kindle, these symbols get converted successfully and are visible in the MOBI file (despite not being visible in the EPUB).

The bottom line is that it would be a good idea to have an option in Sigil that disables conversion of HTML entities and Unicode codes into symbols. Additionally, it would be great if Sigil could show all popular Unicode characters (probably a lot of work for the developers).

Another problem: Sigil generates an HTML table of contents that is single-spaced and inconvenient on smartphones. You need to insert extra spacing by hand to make this table of contents usable.

Another pitfall: A multilevel NCX doesn't work on all Kindle devices and reading apps (especially old Kindle devices). Even in modern reading apps, Kindle for PC shows only the first level, and Kindle for Android supports only two levels. There needs to be some kind of warning in Flight Crew or in help files about this problem, if users decide to convert an EPUB to Kindle. Many users assume that an epubcheck-compliant EPUB is 100% ready for Kindle conversion. In actuality, there are many small pitfalls, listed in my Kindle formatting manual (search for "EPUB" in the ebook).

[Toxaris]

Originally Posted by man2010

Another pitfall: A multilevel NCX doesn't work on all Kindle devices and reading apps (especially old Kindle devices). Even in modern reading apps, Kindle for PC shows only the first level, and Kindle for Android supports only two levels. There needs to be some kind of warning in Flight Crew or in help files about this problem, if users decide to convert an EPUB to Kindle. Many users assume that an epubcheck-compliant EPUB is 100% ready for Kindle conversion. In actuality, there are many small pitfalls, listed in my Kindle formatting manual (search for "EPUB" in the ebook).

Sigil is an ePUB2 editor and does not create Kindle files. Multilevel NCX is perfectly valid ePUB2. If the conversion of a perfectly valid ePUB to a file for Kindle gives issues, it is not the fault of the ePUB or Sigil, but the conversion process. The warnings should be given by the conversion process, not the ePUB editor.

[Toxaris]

Originally Posted by man2010

Another problem: Sigil generates an HTML table of contents that is single-spaced and inconvenient on smartphones. You need to insert extra spacing by hand to make this table of contents usable.

You can add your own stylesheet for the HTML TOC without problems. Just an example is given. If you don't like the standard stylesheet, you are free to adjust it.

[Toxaris]

Originally Posted by man2010

These problems are related to conversion of Sigil-generated EPUB files to Kindle or adaptation of HTML code from a Kindle ebook to EPUB via Sigil.

Sigil usually converts the code of Unicode symbols into symbols themselves in UTF-8 files (when you import an HTML file into Sigil); later, when you try to convert such an EPUB file to Kindle, these symbols may get mangled.

Let's say your HTML file contains this line of code:

⌘ Hello →

After you edit this file in Notepad++ (some other lines of code) and save the file, this line of code will stay the same:

⌘ Hello →

In contrast, if you import this HTML file into Sigil, and then save the EPUB file, this line of code will get converted to:

⌘ Hello →

Thus, Sigil created a possible problem with encoding. There is no way to turn off automatic conversion of HTML entities into symbols in Sigil, except to specify individual HTML entities (inconvenient). Some EPUB readers may support HTML entities but not the resulting symbols. For instance, Internet Explorer (not an EPUB reader) reads HTML entity ’ but cannot read the corresponding symbol in UTF-8 files. There used to be a built-in validation tool in KindleGen 2.3 and 2.4, which produced warnings for various symbols; these warnings disappeared after you replaced the symbols with HTML entities or Unicode codes. This observation leads me to believe that the Kindle platform prefers HTML entities to symbols.

Another example, on another forum, a poster reported that musical score software Finale 2012 produces an EPUB3 file that converts OK to Kindle, but if you import this EPUB file into Sigil, edit, and save it, then the resulting EPUB file cannot be converted to Kindle (most musical symbols get corrupted). This problem existed 3 years ago. During recent testing, I was unable to find examples of such symbols (I don't have Finale 2012). It is possible that this bug was fixed in the latest version of KindleGen. Nevertheless, I found some strange behavior of Sigil with respect to Unicode symbols. If you test this list of symbols:

⌚ ⏛ ⏚ ⏰ ♕ ⛄ ⏳ ☻ ☼ ♪ ⏎ ⏏ ⊄ ⊆ ⊥ ∏ ∉ ⓳

In Sigil, all these pieces of code get automatically (and successfully) converted into corresponding Unicode symbols, except Sigil cannot show many of them (you see a white square in HTML and book views). When you convert the resulting EPUB into Kindle, these symbols get converted successfully and are visible in the MOBI file (despite not being visible in the EPUB).

Whether Sigil can show the Unicode symbol, depends on the font used for displaying and not Sigil. If the symbol is not in the font, it will not be shown.

There are many issues with entities, especially the   is known to give issues. AFAK this has partially to do with BookView. For one thing it depends on the DOCTYPE, which is not required in ePUB (although recommended for ePUB2). The Unicode notation is universal and should be supported by all, even if they are not user/programmer friendly. Again, of course they need to be in the font. No HTML viewer should have issues with this, this should be supported.
Sigil uses the Unicode notation, which will show as the symbol in the editor. If you want to keep seeing the entity, you will need to add it to the preserve entities.
Since Sigil is on the route of supporting ePUB3, the issues with the entities can be solved by using the Unicode notation. Entities are not supported in ePUB3 (or HTML5) anymore.

[rubeus]

Originally Posted by man2010

⌚ ⏛ ⏚ ⏰ ♕ ⛄ ⏳ ☻ ☼ ♪ ⏎ ⏏ ⊄ ⊆ ⊥ ∏ ∉ ⓳

In Sigil, all these pieces of code get automatically (and successfully) converted into corresponding Unicode symbols, except Sigil cannot show many of them (you see a white square in HTML and book views). When you convert the resulting EPUB into Kindle, these symbols get converted successfully and are visible in the MOBI file (despite not being visible in the EPUB).

If a unicode char isn't displayed it is lacking in the used font. Neither SIGIL nor epub2 is responsible and/or accountable for that.

[man2010]

Originally Posted by Toxaris

You can add your own stylesheet for the HTML TOC without problems. Just an example is given. If you don't like the standard stylesheet, you are free to adjust it.

Of course I can edit the stylesheet, but IMO the default needs to be with extra spacing. Most people insert Sigil's HTML table of contents and are unaware that this TOC is unusable on smartphones.

[man2010]

Originally Posted by rubeus

If a unicode char isn't displayed it is lacking in the used font. Neither SIGIL nor epub2 is responsible and/or accountable for that.

Oops, I didn't think of that. Thanks. Perhaps it would be helpful to make the default font the one that shows most of Unicode characters to prevent confusion. Right now the default font in Sigil is Arial, which doesn't show many Unicode characters.

[man2010]

Originally Posted by Toxaris

Sigil is an ePUB2 editor and does not create Kindle files. Multilevel NCX is perfectly valid ePUB2. If the conversion of a perfectly valid ePUB to a file for Kindle gives issues, it is not the fault of the ePUB or Sigil, but the conversion process. The warnings should be given by the conversion process, not the ePUB editor.

I know that multilevel NCX is valid in EPUB. I thought that Sigil developers cared about trouble-free conversion to Kindle. In the past, they ensured that the cover image is not doubled during conversion to Kindle, and they removed the "text" item from the giude section of the default OPF file. Perhaps mentioning this issue with NCX in the documentation would be useful.

[man2010]

Originally Posted by Toxaris

The Unicode notation is universal and should be supported by all, even if they are not user/programmer friendly. Again, of course they need to be in the font. No HTML viewer should have issues with this, this should be supported.
Sigil uses the Unicode notation, which will show as the symbol in the editor. If you want to keep seeing the entity, you will need to add it to the preserve entities.
Since Sigil is on the route of supporting ePUB3, the issues with the entities can be solved by using the Unicode notation. Entities are not supported in ePUB3 (or HTML5) anymore.

What do you mean by "Unicode notation"?

Why does Sigil have to convert all codes and entities into symbols in the code view?

[DiapDealer]

Originally Posted by man2010

Why does Sigil have to convert all codes and entities into symbols in the code view?

Because Gumbo (the Sigil html parsing engine) does not grok entities. You can, however, enter a list of entities you wish to preserve in the Preserve Entities preferences (Edit->Preferences->Preserve Entities).

As for some sort of warning that an epub is not 100% ready to be converted to a Kindlebook ... that's just not going to be an integral feature of Sigil. Someone could write a validation plugin that could check for common kindle conversion bug-bears, though. There could even be an output plugin that fixes some of those bugbears. In fact; almost all of the points you raise sound like perfect examples of where plugins can come the rescue.

Originally Posted by man2010

Sigil usually converts the code of Unicode symbols into symbols themselves in UTF-8 files (when you import an HTML file into Sigil); later, when you try to convert such an EPUB file to Kindle, these symbols may get mangled.

I guess I'd need to see some specific examples of where/when this "mangling" occurs. Because if a Kindle can properly display the numerical/named entity, then it should have absolutely no problem displaying the character it represents. Either the necessary glyph is included in the Kindle's fonts, or they're not. I fail to see how Sigil's conversion of entities to characters can be the cause of any improper Kindle rendering. But again... you could always preserve a list entities that you deem to be problematic on Kindles if you like--though it's my contention that as far as Kindles go: what's good for the entity is good for the char. Numerical or named entities don't magically create renderable glyphs out of thin air, after all.

[Notjohn]

Originally Posted by man2010

I thought that Sigil developers cared about trouble-free conversion to Kindle. In the past, they ensured that the cover image is not doubled during conversion to Kindle, and they removed the "text" item from the giude section of the default OPF file.

That was never my understanding.... Well, the developers may care about the conversion, but I don't think they did anything to smooth it. It was Amazon that opened its KDP platform to epub format, and things that Amazon did (or didn't do) that occasionally doubled the cover and that sets the Start Reading Location.

(Good to see you on this forum, Man2010!)

[DiapDealer]

Put it this way: we love the idea of a trouble-free conversion of a Sigil created epub to a Kindlebook. We're not, however, likely to help accomodate the quirks of such a conversion when those quirks involve a deviation from the epub specifications.

In my opinion, any reading system that will render an entity (numeric, hex, or named) properly, but not its character representation is simply broken (epub or otherwise). Glyphs are glyphs.

Cases where neither the entity NOR the character will display are either a case of the system/reader font missing the necessary glyphs, or a case of character encoding conflicts. The former is out of our control, and the latter is something we will not tackle. Epub uses utf8/16. Period. Anything opened in Sigil will be forced into that particular character encoding box.

[theducks]

Originally Posted by man2010

Of course I can edit the stylesheet, but IMO the default needs to be with extra spacing. Most people insert Sigil's HTML table of contents and are unaware that this TOC is unusable on smartphones.

A phone is not what EPUB2 was designed against.
And the NCX is the official EPUB2 way.



Make your modified sigil_toc.css and simply replace the one Sigil creates if that floats your phone.
Sigil WILL NOT replace this file if you later regenerate the TOC

[Turtle91]

Originally Posted by man2010

Of course I can edit the stylesheet, but IMO the default needs to be with extra spacing. Most people insert Sigil's HTML table of contents and are unaware that this TOC is unusable on smartphones.

You can edit the stylesheet each time...or...as ducks mentioned, you can save your stylesheet and Sigil will use your stylesheet whenever it creates a new TOC.

Also, just because a reader is using a smartphone does not make the TOC unusable. It depends entirely on the app you are using. If your app doesn't support multi-level TOC/NCX then change apps, or complain to the maker of your app. Unfortunately, if you are trying to convert things to kindle format then you are using Amazon and your complaints will fall on deaf ears. IMO, the kindle app sucks. I would recommend using an app - Marvin for iOS, or any number of apps on android - that will read your ePub directly. Then you will not have any conversion or TOC issues.

Cheers!

[DiapDealer]

Originally Posted by man2010

Another pitfall: A multilevel NCX doesn't work on all Kindle devices and reading apps (especially old Kindle devices). Even in modern reading apps, Kindle for PC shows only the first level, and Kindle for Android supports only two levels. There needs to be some kind of warning in Flight Crew or in help files about this problem, if users decide to convert an EPUB to Kindle. Many users assume that an epubcheck-compliant EPUB is 100% ready for Kindle conversion. In actuality, there are many small pitfalls, listed in my Kindle formatting manual (search for "EPUB" in the ebook).

In my opinion, this is not within Sigil's purview. Nor is it Flightcrew's. It is up to the Kindlebook creator to learn what is necessary to make an epub "ready" to be converted to a Kindlebook. It is up to Sigil to make a valid epub.

We provide tools (and a plugin framework) that make it possible to make epubs that make working Kindlebooks (for instance, the ncx/toc can easily be "flattened" using the gui ToC editor) painless and smooth. But we're not here to teach Kindlebook creator's their craft. We're here to provide epub creators with the tools to create epubs.