Non-displayable characters

GregTheGrate · 04-09-2014, 03:23 AM

I recently downloaded a book with non-displayable characters in it. Kindle White shows the Unicode in a box. Unfortunately I couldn't see these codes in the editor and had to use Sigil to fix the text.

The Calibre editor seemed to have ignored the non-displayable characters in both panes.

Is there an option I should be setting so that I can see these letters so that I am able to replace them. They are typlically the left and right-quotes, apostrophe and em-dash.

Sigil displays them as a question mark in a box.

Many thanks

DoctorOhh · 04-09-2014, 06:42 AM

Quote:

Originally Posted by GregTheGrate

Sigil displays them as a question mark in a box.

The calibre editor marks them by highlighting them in yellow. If you put your cursor to the right of the character calibre will display what the character is in the lower right hand corner. Although why you cannot see the characters you cited is beyond me.

theducks · 04-09-2014, 09:11 AM

Quote:

Originally Posted by DoctorOhh

The calibre editor marks them by highlighting them in yellow. If you put your cursor to the right of the character calibre will display what the character is in the lower right hand corner. Although why you cannot see the characters you cited is beyond me.

The only reason I can figure is the CURRENT font set has been stripped(subseted) or never had those glyphs

There will be 2 font sets involved:
1) the font used for Code view
2) the embedded font for book view (I think that this one only applies if there is a @font setting), otherwise: the font-family dishes what is available

mrmikel · 04-09-2014, 09:20 AM

Both editor and preview fonts are set in Preferences. These particular characters should show. Change the fonts shown there to something else and see how it looks.

GregTheGrate · 04-09-2014, 07:25 PM

I think I may have mislead you. In their normal form, the quotation marks, apostrophes and em dash characters all show. However, when I downloaded the book, the left quotation was shown as Unicode 0082 and the right as 0083. These are non-displayable characters in any font, but were consistently used throughout the book to represent quotation marks.

Therefore, I need to replace these 0082 and 0083 characters with “ and ” respectively.

If I use Calibre Editor to see the downloaded text, none of the 0082 or 0083 characters are visible. I have just determined that a space is displayed, which one can copy and search for.

In Sigil, they are displayed as question-mark encased in a square.

It is these non-displayable characters for which I am seeking advice.

Sorry for the confusion

mrmikel · 04-09-2014, 08:32 PM

You can enter the unicode in regex and find them that way. You'd need the unicode number to search for them. There are many tables on the web, and maybe even on this forum which will tell you...Actually, in the special characters of calibre, it will tell you what the number is.

I think you might be able to use the special characters function of the editor to enter them in find.

Divingduck · 04-11-2014, 10:30 AM

Isn't it possible to do a search & replace?

Search for: \u0082
Replace with: “

For seeing the characters in the code view you maybe need to change the preferences for the editor font family to a font what includes the characters you need (for Windows something like Arial Unicode MS). I use this font, because it has the most extensive font set.

GregTheGrate · 04-11-2014, 07:24 PM

Many thanks to all for the editing suggestions. The problem though is not editing, but seeing the non-displayable characters even if only in code view. They are not visible. They show as spaces.

I tried Divingduck's font-change suggestion - even though I run on a Mac. Unfortunately the existence of the characters is still not visible.

Perhaps Kovid might comment here?

roger64 · 04-11-2014, 08:51 PM

Hi

Quote:

Originally Posted by GregTheGrate

Many thanks to all for the editing suggestions. The problem though is not editing, but seeing the non-displayable characters even if only in code view. They are not visible. They show as spaces.

I tried too to learn more about "invisible" characters (i.e. non displayed in code view) by looking for two former entities nbsp (visible in Code view with an orange rectangle) and nnbsp (this one invisible). Note that the latter is displayed in Pre-view and that in Code view you can "feel" them with the cursor and get information with the lower right field.

This is what I found.
https://www.mobileread.com/forums/sho...4&postcount=11

The only - clumsy - way today to make this invisible character visible in Code view would be to search and replace it by the entity code &# 8239; (without space). Only then would the invisible" nnbsp be displayed by its entity code name &# 8239; in code view in the calibre editor. But you would have also to insert the DOCTYPE required for entities... I do not think this would be a good solution.

kovidgoyal · 04-11-2014, 09:28 PM

U+0082 is not an opening or a closing quote of any kind. It is an ascii control code (Break permitted here). Being invisible is the correct behavior for it. There are almost no fonts that render it. For a list see here: http://www.fileformat.info/info/unic...r/82/index.htm

If you have it in your book, it is almost certainly the result of incorrect character decoding. Simply use search and replace to replace it with the correct character.

Divingduck · 04-12-2014, 02:30 AM

Ups, sorry. I had not looked to the code. As Kovid is suggesting, replacing is the best way to solve it.

@Kovid,
Only an idea. I know your solution is correct, but is it maybe possible, to render not included Unicode characters in code view window with a replacement placeholder? Maybe in combination with a switch to use it only if needed.

kovidgoyal · 04-12-2014, 02:59 AM

@Divingduck: That is not possible to do without re-implementing the Qt text rendering stack, which is waaaay too much work.

Divingduck · 04-12-2014, 08:12 AM

I think, it is better to forget my idea. There are much more important things what are waiting for your implementation

04-09-2014, 03:23 AM	#1
GregTheGrate Connoisseur Posts: 62 Karma: 10 Join Date: Jun 2012 Device: Kindle, iPad	Non-displayable characters I recently downloaded a book with non-displayable characters in it. Kindle White shows the Unicode in a box. Unfortunately I couldn't see these codes in the editor and had to use Sigil to fix the text. The Calibre editor seemed to have ignored the non-displayable characters in both panes. Is there an option I should be setting so that I can see these letters so that I am able to replace them. They are typlically the left and right-quotes, apostrophe and em-dash. Sigil displays them as a question mark in a box. Many thanks

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
How many characters will you use?	frahse	Writers' Corner	21	09-21-2012 10:44 AM
¿Convert unicode decomposed characters to unique/normal characters?	JohnQwerty	Calibre	3	04-05-2012 12:08 PM
All characters are fictitious ...	mr ploppy	Writers' Corner	8	07-19-2011 05:04 AM
Classic using characters on NOOK	nmed	Barnes & Noble NOOK	1	08-02-2010 06:09 PM
Characters missing	orailean	Calibre	2	07-23-2008 07:02 AM

04-09-2014, 09:20 AM	#4
mrmikel Color me gone Posts: 2,089 Karma: 1445295 Join Date: Apr 2008 Location: Central Oregon Coast Device: PRS-300	Both editor and preview fonts are set in Preferences. These particular characters should show. Change the fonts shown there to something else and see how it looks.

04-09-2014, 07:25 PM	#5
GregTheGrate Connoisseur Posts: 62 Karma: 10 Join Date: Jun 2012 Device: Kindle, iPad	I think I may have mislead you. In their normal form, the quotation marks, apostrophes and em dash characters all show. However, when I downloaded the book, the left quotation was shown as Unicode 0082 and the right as 0083. These are non-displayable characters in any font, but were consistently used throughout the book to represent quotation marks. Therefore, I need to replace these 0082 and 0083 characters with “ and ” respectively. If I use Calibre Editor to see the downloaded text, none of the 0082 or 0083 characters are visible. I have just determined that a space is displayed, which one can copy and search for. In Sigil, they are displayed as question-mark encased in a square. It is these non-displayable characters for which I am seeking advice. Sorry for the confusion

04-09-2014, 08:32 PM	#6
mrmikel Color me gone Posts: 2,089 Karma: 1445295 Join Date: Apr 2008 Location: Central Oregon Coast Device: PRS-300	You can enter the unicode in regex and find them that way. You'd need the unicode number to search for them. There are many tables on the web, and maybe even on this forum which will tell you...Actually, in the special characters of calibre, it will tell you what the number is. I think you might be able to use the special characters function of the editor to enter them in find.

04-11-2014, 10:30 AM	#7
Divingduck Wizard Posts: 1,166 Karma: 1410083 Join Date: Nov 2010 Location: Germany Device: Sony PRS-650	Isn't it possible to do a search & replace? Search for: \u0082 Replace with: “ For seeing the characters in the code view you maybe need to change the preferences for the editor font family to a font what includes the characters you need (for Windows something like Arial Unicode MS). I use this font, because it has the most extensive font set.

04-11-2014, 07:24 PM	#8
GregTheGrate Connoisseur Posts: 62 Karma: 10 Join Date: Jun 2012 Device: Kindle, iPad	Many thanks to all for the editing suggestions. The problem though is not editing, but seeing the non-displayable characters even if only in code view. They are not visible. They show as spaces. I tried Divingduck's font-change suggestion - even though I run on a Mac. Unfortunately the existence of the characters is still not visible. Perhaps Kovid might comment here?

04-11-2014, 09:28 PM	#10
kovidgoyal creator of calibre Posts: 45,257 Karma: 27110894 Join Date: Oct 2006 Location: Mumbai, India Device: Various	U+0082 is not an opening or a closing quote of any kind. It is an ascii control code (Break permitted here). Being invisible is the correct behavior for it. There are almost no fonts that render it. For a list see here: http://www.fileformat.info/info/unic...r/82/index.htm If you have it in your book, it is almost certainly the result of incorrect character decoding. Simply use search and replace to replace it with the correct character.

04-12-2014, 02:30 AM	#11
Divingduck Wizard Posts: 1,166 Karma: 1410083 Join Date: Nov 2010 Location: Germany Device: Sony PRS-650	Ups, sorry. I had not looked to the code. As Kovid is suggesting, replacing is the best way to solve it. @Kovid, Only an idea. I know your solution is correct, but is it maybe possible, to render not included Unicode characters in code view window with a replacement placeholder? Maybe in combination with a switch to use it only if needed.

04-12-2014, 02:59 AM	#12
kovidgoyal creator of calibre Posts: 45,257 Karma: 27110894 Join Date: Oct 2006 Location: Mumbai, India Device: Various	@Divingduck: That is not possible to do without re-implementing the Qt text rendering stack, which is waaaay too much work.

04-12-2014, 08:12 AM	#13
Divingduck Wizard Posts: 1,166 Karma: 1410083 Join Date: Nov 2010 Location: Germany Device: Sony PRS-650	I think, it is better to forget my idea. There are much more important things what are waiting for your implementation

Advert

Advert