nbsp disappeared

[BobC]

A bit more on this - I looked at the Issues log for Sigil and discovered the following under Issue 1791: All   are replaced by normal space when switching from Book View to Code View: back in November 2012:

Project Member #3 john@nachtimwald.com
......
Are the   characters being changed to their unicode equivalents which appear the same as a space but are a different code point? I believe there was code in 0.5.3 that converted some characters to entities that was removed due to user demand.

So it looks like Sigil used to do the conversion but this was removed.

Looking at it logically it really isn't Sigil's job to convert things unless asked to. It's an editor.

BobC

[roger64]

Originally Posted by BobC

A bit more on this - I looked at the Issues log for Sigil and discovered the following under Issue 1791: All   are replaced by normal space when switching from Book View to Code View: back in November 2012:

Project Member #3 john@nachtimwald.com
......
Are the   characters being changed to their unicode equivalents which appear the same as a space but are a different code point? I believe there was code in 0.5.3 that converted some characters to entities that was removed due to user demand.

So it looks like Sigil used to do the conversion but this was removed.

Looking at it logically it really isn't Sigil's job to convert things unless asked to. It's an editor.

BobC

Thank you BobC. Well we can say that we have the full and clear story now.

I am a longtime Sigil user stricken by a triple curse:
- I use odt and their conversion libraries seem to be not up-to standard (see the previous BobC message)
- I am a Linux user and I needed to wait for DiapDealer deb to get a decent 0.7.2 version (this to explain why my bug report came so late)
- last but not least, I am French and makes extensive use of &nbsp for legitimate typesetting reasons (hello Jellby!) contrary to many English language users who consider them just to be some parasite to eradicate.

And now, I should use systematically an old Sigil to save first my EPUBs just for keeping alive the numerous   I converted from my odt files.

That's a bit too much.

Appeal to Editor's Editor.

[funeral music here]
Do consider please how many silent and misunderstood mistakes have been triggered by this behaviour for users of Calibre, writer2xhtml, writer2epub, and most probably other odt users (Amanuensis, ...), who also happen to use your wonderful but sometimes tricky software.

Every time they tried to use your Editor to correct/enhance some EPUBs from these converters (and this happened for me more than a couple of times for French books), unknowingly they slaughtered their innocent   poor victims of this cruel and silent serial killer.

For all these reasons, also for the sake of humanity, of pity, of whatever you call it, please restore Sigil abilities to its former status on   This is a user demand, a badly mistreaten one at that...
[end of funeral music]

[PeterT]

Roger64; I think some of my confusion in following this has been the usage by you of the term &nbsp throughout. When I (and others) looked in your document there were non period. Launching Libre Office (sorry I don't have Open Office installed) I see the option exists to "insert a non-breaking-space", but from what I have subsequently learnt, this is NOT saved as the sequence &nbsp, bur rather as a unicode character.

[roger64]

Originally Posted by PeterT

Roger64; I think some of my confusion in following this has been the usage by you of the term &nbsp throughout. When I (and others) looked in your document there were non period. Launching Libre Office (sorry I don't have Open Office installed) I see the option exists to "insert a non-breaking-space", but from what I have subsequently learnt, this is NOT saved as the sequence &nbsp, bur rather as a unicode character.

Thanks for this info. I published my source odt file, the one I used with writer2xhtml and Calibre to create my EPUBs.

I am a longtime OpenOffice user. To insert a   I first suppress the former whitespace it replaces, then use the command Shift-Control-Space and this creates a greyed out area which signify there is (or should be) a   here. There are also typographical utilities around that do the same job for you following French typographic rules.

I did not know about your point. To expect the average user to be knowledgeable about this subtility is a dream. I think it would be fair not to count on it and help him supplying a   where he asks for an  .

[DiapDealer]

Be sure to check your clean source settings. Setting it to clean upon "Open" just may restore the functionality you're accustomed to.

I think it's a little confusing because people are thinking that entities are disappearing, when in fact, their are no actual nbsp entities in the odt output. Only special characters that some software has been known to silently convert to a named entity. For better or worse (individuals will have to decide for themselves), Sigil is moving toward NOT making any silent changes to code.

But do try the different clean source settings of Sigil to see if they have any bearing on your situation.

[roger64]

Originally Posted by DiapDealer

Be sure to check your clean source settings. Setting it to clean upon "Open" just may restore the functionality you're accustomed to.

Yes, I'll do it and report asap. Up to now, I am sure Sigil does NOT save on opening and closes (I unticked this in Preferences). It saves the EPUB only when I ask it to do so.

On writer2xhtml options, there is a mysterious line -for me- about named entities. Do you think it could be good to tick it? (see arrow on screenshot).

For better or worse (individuals will have to decide for themselves), Sigil is moving toward NOT making any silent changes to code.

I have already experimented this kind of situation for straight and curled apostrophes. Originally, the spellchecker flagged all the words including a curled apostrophe until I made the point that French users replace systematically straight by curled and that it made the spellchecker unusable for them. Now this question has been beautifully solved, I could not precisely explain how.

So, if we explain slowly the problems this behaviour causes for French (and probably some other languages), maybe we could see appear a Unicode/  option to tick somewhere. One Editor's purpose is primarily to help people and not to make things worse... To lose for good 2 500   precisely placed, is an unpleasant thing...

[meme]

Originally Posted by roger64

On writer2xhtml options, there is a mysterious line -for me- about named entities. Do you think it could be good to tick it? (see arrow on screenshot).

You can try it, but at least when I tested it did not help, even though it seems like it should.

First, thanks for the epub file. It clearly shows that the nbsp characters are in the file before being opened by Sigil and provided something to test with.

The issue appears (pending further checking) to be with Qt5 (not much of a surprise). Possibly a change from Qt4 to Qt5.

The file is encoded okay as UTF-8, and the nbsp characters are in the file stored as two bytes C2 A0 (or 302 240 decimal) as you can see when you examine the file with a hex editor. When Sigil opens the file it correctly identifies the file as UTF-8 and then asks Qt to convert the file to Unicode. Unfortunately for some reason, although it appears to convert everything else ok, Qt converts the 2 bytes to a standard space (20) instead of the nbsp character (A0). So the nbsp characters are removed before the rest of Sigil can see them and convert them to the   entity.

A short test shows that we can make a specific check for this 2 byte pair in UTF-8 files and map them to   before doing the conversion to unicode, thus preserving the nbsps in the file. Although it needs to be checked to make sure it doesn't break anything else, and to see if there is a better workaround.

[roger64]

Outputting odt with named entities

I tried to convert my odt file using named entities. My converter (writer2xhtml) allows me to do that. Now Sigil 0.7.2 is very happy and display the real   that he finds immediately on the EPUB. However, there is a drawback...
é is now &eacute
French quotes become &laquo and &raquo
and so on....

See screenshot.

The text in Book-view is fine, but in Code view, it's very difficult to read and furthermore these entities fool totally the spellchecker.

So, the advised medecine seems to be worse than the illness. We cannot honestly recommend to output odt using named entities, because of these kind of problems, at least in French where we have lots of accents.

[Toxaris]

I would definetly use that option. Officially in ePUB3 named entities are no longer supported, but that is a long way still. Even so, look at other deprecated options in HTML, they almost all still work.
It would at least be a lot easier to see if the nbsp will be in the source then.

[roger64]

@meme

Thanks very much for your detailed reply and taking care of this.

[meme]

Interesting that your test worked - guess I either have a different version of writer2xhtml or I didn't set it up right.

As a workaround you can use the writer2xhtml option so that named entities are used, and then you can create a Save Search group to replace each named entity with the actual character (e.g. replace &eactute; with é). Not ideal, but until there is a fix it seems the best option.

The   entity will always show up in Code View, even if this issue is fixed. This is because if the nbsp character is used, the Qt based Book View would incorrectly convert it to a normal space.

[roger64]

@meme

Yes I am used to see the   entity in Code view. Usually, I later change most of these   (2/3?) to nnbsp which are typographically nicer and do not show up in CodeView. (&-#8239; )

It could be possible to change all the named entities with Saved Search but I have an easier -for me- solution. While waiting for a fix, I will rather save first the EPUB with Sigil 0.5.3. That will create real   .

Knowing that a fix will come change the way I feel about it.

[BobC]

@roger64

Slightly off-topic but what version of Libre Office are you using ? I tried to install the writer2html extension on my LO 4.0.2.2 and it fails with an error message "Could not create Java implementation loader".

I'm guessing this is because writer2html has not been updated to work with LO 4.

BobC

[roger64]

Originally Posted by BobC

@roger64

Slightly off-topic but what version of Libre Office are you using ? I tried to install the writer2html extension on my LO 4.0.2.2 and it fails with an error message "Could not create Java implementation loader".

I'm guessing this is because writer2html has not been updated to work with LO 4.

BobC

I use -a little- LibreOffice 4.0.2 and still mostly ApacheOpenOffice 3.4.1, both with writer2xhtml.

However, as you realized, writer2xhtml has a Java problem with this named version of Libo, due to a deprecated Java bit in its code. A patch to writer2xhtml code has been produced and I published a patched version of this extension four weeks ago.

You will find relevant information on this French thread (but the patch info is also in English )

[BobC]

@roger64

Thank you for the patched version - it didn't work for me at first but then I realised my JRE was broken. Reinstalled JRE 7 and all is fine.

It took me some time to discover how to configure the export filter to use HTML entities such as   but now I have found it I'll be using this as my usual way to generate epubs from ODTs.

BobC