Non-displayable characters

[GregTheGrate]

I recently downloaded a book with non-displayable characters in it. Kindle White shows the Unicode in a box. Unfortunately I couldn't see these codes in the editor and had to use Sigil to fix the text.

The Calibre editor seemed to have ignored the non-displayable characters in both panes.

Is there an option I should be setting so that I can see these letters so that I am able to replace them. They are typlically the left and right-quotes, apostrophe and em-dash.

Sigil displays them as a question mark in a box.

Many thanks

[DoctorOhh]

Originally Posted by GregTheGrate

Sigil displays them as a question mark in a box.

The calibre editor marks them by highlighting them in yellow. If you put your cursor to the right of the character calibre will display what the character is in the lower right hand corner. Although why you cannot see the characters you cited is beyond me.

[theducks]

Originally Posted by DoctorOhh

The calibre editor marks them by highlighting them in yellow. If you put your cursor to the right of the character calibre will display what the character is in the lower right hand corner. Although why you cannot see the characters you cited is beyond me.

The only reason I can figure is the CURRENT font set has been stripped(subseted) or never had those glyphs

There will be 2 font sets involved:
1) the font used for Code view
2) the embedded font for book view (I think that this one only applies if there is a @font setting), otherwise: the font-family dishes what is available

[mrmikel]

Both editor and preview fonts are set in Preferences. These particular characters should show. Change the fonts shown there to something else and see how it looks.

[GregTheGrate]

I think I may have mislead you. In their normal form, the quotation marks, apostrophes and em dash characters all show. However, when I downloaded the book, the left quotation was shown as Unicode 0082 and the right as 0083. These are non-displayable characters in any font, but were consistently used throughout the book to represent quotation marks.

Therefore, I need to replace these 0082 and 0083 characters with “ and ” respectively.

If I use Calibre Editor to see the downloaded text, none of the 0082 or 0083 characters are visible. I have just determined that a space is displayed, which one can copy and search for.

In Sigil, they are displayed as question-mark encased in a square.

It is these non-displayable characters for which I am seeking advice.

Sorry for the confusion

[mrmikel]

You can enter the unicode in regex and find them that way. You'd need the unicode number to search for them. There are many tables on the web, and maybe even on this forum which will tell you...Actually, in the special characters of calibre, it will tell you what the number is.

I think you might be able to use the special characters function of the editor to enter them in find.

[Divingduck]

Isn't it possible to do a search & replace?

Search for: \u0082
Replace with: “

For seeing the characters in the code view you maybe need to change the preferences for the editor font family to a font what includes the characters you need (for Windows something like Arial Unicode MS). I use this font, because it has the most extensive font set.

[GregTheGrate]

Many thanks to all for the editing suggestions. The problem though is not editing, but seeing the non-displayable characters even if only in code view. They are not visible. They show as spaces.

I tried Divingduck's font-change suggestion - even though I run on a Mac. Unfortunately the existence of the characters is still not visible.

Perhaps Kovid might comment here?

[roger64]

Hi

Originally Posted by GregTheGrate

Many thanks to all for the editing suggestions. The problem though is not editing, but seeing the non-displayable characters even if only in code view. They are not visible. They show as spaces.

I tried too to learn more about "invisible" characters (i.e. non displayed in code view) by looking for two former entities nbsp (visible in Code view with an orange rectangle) and nnbsp (this one invisible). Note that the latter is displayed in Pre-view and that in Code view you can "feel" them with the cursor and get information with the lower right field.

This is what I found.
https://www.mobileread.com/forums/sho...4&postcount=11

The only - clumsy - way today to make this invisible character visible in Code view would be to search and replace it by the entity code &# 8239; (without space). Only then would the invisible" nnbsp be displayed by its entity code name &# 8239; in code view in the calibre editor. But you would have also to insert the DOCTYPE required for entities... I do not think this would be a good solution.

[kovidgoyal]

U+0082 is not an opening or a closing quote of any kind. It is an ascii control code (Break permitted here). Being invisible is the correct behavior for it. There are almost no fonts that render it. For a list see here: http://www.fileformat.info/info/unic...r/82/index.htm

If you have it in your book, it is almost certainly the result of incorrect character decoding. Simply use search and replace to replace it with the correct character.

[Divingduck]

Ups, sorry. I had not looked to the code. As Kovid is suggesting, replacing is the best way to solve it.

@Kovid,
Only an idea. I know your solution is correct, but is it maybe possible, to render not included Unicode characters in code view window with a replacement placeholder? Maybe in combination with a switch to use it only if needed.

[kovidgoyal]

@Divingduck: That is not possible to do without re-implementing the Qt text rendering stack, which is waaaay too much work.

[Divingduck]

I think, it is better to forget my idea. There are much more important things what are waiting for your implementation