Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Editor

Notices

Reply
 
Thread Tools Search this Thread
Old 04-30-2018, 06:59 PM   #1
G2B
Enthusiast
G2B began at the beginning.
 
Posts: 28
Karma: 10
Join Date: Feb 2018
Device: PC / iPad
This is so weird.

I use latest 64-bit version of Calibre on Win10 PC.

I've known for a while that if you get a parsing error (tag imbalance) that sometimes a black diamond with question mark shows in the text. This does not happen with every parsing problem, not sure what causes this to appear.
This time, I had something really weird happen, that I've never seen before.

The file I was practicing with obviously had some processing problems from previous editor: some word spacings were deleted where they shouldn't and two words were stuck together in many locations. I only noticed that this early in the editing process (spelling check usually is my last step) because that is where the black diamonds showed up.

What caused the black diamonds to appear: I deleted the initial span-tag at the top of the page. This caused parsing error, and the diamonds appeared (Pic1)

The black diamonds showed only in the preview screen, not in the editor screen. Because the diamonds do not appear in the editor screen, I cannot search for them. When I find one in preview and push delete button in the corresponding editor position, the cursor does not advance, but the black diamond in preview disappears.


Even stranger, when I used 'auto correct errors' (which just deletes the superfluous end tag and removes the error, which normally makes the diamonds disappear) the diamonds not only stayed in preview, but also appeared in the editor (Pic2) And because they did show in the editor now; I was able to use the editor to get rid of them all at once.

I have attached the file that showed this problem. It is an incomplete (less than 1 chapter) extract from a book that I use for self-education practice with the editor. The diamonds appear by removing the initial span tag after the title.
They stay even when the final closing tag is removed.

QUESTIONS:
1. Why do those diamonds only appear in a (very small) minority of files with parsing errors, not in others even though they may have similar errors.

2. What's up with the invisible character? Before today, if they showed, these diamonds always showed in both editor and preview screens. This is new. Not only invisible, but also not taking up any space in the editor? Is that possibly a bug?

Comments invited and appreciated.
Attached Thumbnails
Click image for larger version

Name:	weird (1).jpg
Views:	387
Size:	244.1 KB
ID:	163764   Click image for larger version

Name:	weird (2).jpg
Views:	357
Size:	243.8 KB
ID:	163765  
Attached Files
File Type: epub IoM1.epub (89.7 KB, 224 views)
G2B is offline   Reply With Quote
Old 04-30-2018, 07:21 PM   #2
theducks
Well trained by Cats
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 29,689
Karma: 54369090
Join Date: Aug 2009
Location: The Central Coast of California
Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A
The diamonds are Character unknown errors.
Translation: The coding says use Character set X, but the content was done with Z

This issue is common with older versions of Word and documents done in different countries.

Commercial books should never have this error by the time they are sold.
theducks is offline   Reply With Quote
Advert
Old 04-30-2018, 08:36 PM   #3
retiredbiker
Addict
retiredbiker ought to be getting tired of karma fortunes by now.retiredbiker ought to be getting tired of karma fortunes by now.retiredbiker ought to be getting tired of karma fortunes by now.retiredbiker ought to be getting tired of karma fortunes by now.retiredbiker ought to be getting tired of karma fortunes by now.retiredbiker ought to be getting tired of karma fortunes by now.retiredbiker ought to be getting tired of karma fortunes by now.retiredbiker ought to be getting tired of karma fortunes by now.retiredbiker ought to be getting tired of karma fortunes by now.retiredbiker ought to be getting tired of karma fortunes by now.retiredbiker ought to be getting tired of karma fortunes by now.
 
retiredbiker's Avatar
 
Posts: 378
Karma: 1638210
Join Date: May 2013
Location: Ontario, Canada
Device: Kindle KB, Oasis, Ubuntu, Jutoh,Kobo Forma
Odd Charavters

Yes, an unknown character from bad coding. Others that I have seen fairly frequently are:

(well I pasted in a few, but they were translated and did not show as pasted!)

The black diamond in your case is likely an m-dash, and you can do search and replace to fix it.

But I also do not understand why they sometimes show in the editor panel and other times not -- and why getting rid of a <span> or <div> can make them appear. Anybody know?

Last edited by retiredbiker; 04-30-2018 at 08:41 PM. Reason: Strange characters pasted in were translated...
retiredbiker is offline   Reply With Quote
Old 04-30-2018, 08:43 PM   #4
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 9,864
Karma: 13806776
Join Date: Feb 2009
Location: North Carolina
Device: Icarus Illumina XL HD, Nexus 7
Quote:
Originally Posted by G2B View Post
1. Why do those diamonds only appear in a (very small) minority of files with parsing errors, not in others even though they may have similar errors.

2. What's up with the invisible character? Before today, if they showed, these diamonds always showed in both editor and preview screens. This is new. Not only invisible, but also not taking up any space in the editor? Is that possibly a bug?
I'm not an editing guru, but the diamonds you see are caused by your system not being able to interpret an em dash (—). Someone else will have to explain how to eliminate or replace them. I can see them in the editing window and easily replace them with -- but if you can't see them...
DoctorOhh is offline   Reply With Quote
Old 04-30-2018, 09:24 PM   #5
jackie_w
Grand Sorcerer
jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.
 
Posts: 6,171
Karma: 16228536
Join Date: Sep 2009
Location: UK
Device: Kobo: KA1, ClaraHD, Forma, Libra2, Clara2E. PocketBook: TouchHD3
I opened the test file in the calibre Editor and was not able to see black diamonds in Preview. They were also invisible in Code View but it doesn't mean they aren't there.

Using the first occurrence shown in image 1 in post #1:
- select the dl of what looks like madlike in the first paragraph and press Ctrl F to do a search.
- carefully remove the d and the l which leaves just the middle single "invisible" unknown char in the Find box
- enter an mdash in the Replace box and do a Replace All. 7 occurrences were replaced in the test epub.
jackie_w is offline   Reply With Quote
Advert
Old 04-30-2018, 09:30 PM   #6
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,776
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
There are control codes in that text, probably there as a result of an incorrect conversion from some other encoding to UTF-8 at some time in the past. In this case the character is https://www.fileformat.info/info/uni...0097/index.htm

This is not a renderable character, which is why it does not show up in the editor. IIRC whether it renders in the preview or not depends on the properties of the font being used there, whether it define s a glyph for this codepoint or not.
kovidgoyal is offline   Reply With Quote
Old 04-30-2018, 10:34 PM   #7
davidfor
Grand Sorcerer
davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.
 
Posts: 24,908
Karma: 47303748
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
On top of all the above, it is probably a good idea to find out what the character is. To do this, put the cursor in the editor window immediately after the character and look in the bottom right of the screen. Calibre will display the name and unicode for the character. In this cases like this, I find it easier to put the the cursor a character or two before the strange character and use the arrow keys to move through them until you find the invisible character.
davidfor is offline   Reply With Quote
Old 05-01-2018, 12:56 AM   #8
BetterRed
null operator (he/him)
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 20,458
Karma: 26645808
Join Date: Mar 2012
Location: Sydney Australia
Device: none
Quote:
Originally Posted by davidfor View Post
On top of all the above, it is probably a good idea to find out what the character is. To do this, put the cursor in the editor window immediately after the character and look in the bottom right of the screen. Calibre will display the name and unicode for the character. In this cases like this, I find it easier to put the the cursor a character or two before the strange character and use the arrow keys to move through them until you find the invisible character.
That was my first thought, but nothing gets displayed, when I looked into Reports and saw U'0097 was nameless I moved on - figuring KG would 'sort it out'

BR
Attached Thumbnails
Click image for larger version

Name:	1.jpg
Views:	316
Size:	220.3 KB
ID:	163769  

Last edited by BetterRed; 05-01-2018 at 01:00 AM. Reason: add attachment
BetterRed is online now   Reply With Quote
Old 05-01-2018, 02:45 AM   #9
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 9,864
Karma: 13806776
Join Date: Feb 2009
Location: North Carolina
Device: Icarus Illumina XL HD, Nexus 7
Quote:
Originally Posted by BetterRed View Post
That was my first thought, but nothing gets displayed, when I looked into Reports and saw U'0097 was nameless I moved on - figuring KG would 'sort it out'
Thanks for pointing me to reports. I couldn't figure out where to find its code.

I wonder why I see it in the editor section and reports, but not the preview window?

If I search for \u0097 in Regex mode it matches the character.
Attached Thumbnails
Click image for larger version

Name:	reports.jpg
Views:	280
Size:	157.0 KB
ID:	163770   Click image for larger version

Name:	search.jpg
Views:	276
Size:	18.1 KB
ID:	163771  

Last edited by DoctorOhh; 05-01-2018 at 03:04 AM.
DoctorOhh is offline   Reply With Quote
Old 05-01-2018, 03:18 AM   #10
davidfor
Grand Sorcerer
davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.
 
Posts: 24,908
Karma: 47303748
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
Quote:
Originally Posted by BetterRed View Post
That was my first thought, but nothing gets displayed, when I looked into Reports and saw U'0097 was nameless I moved on - figuring KG would 'sort it out'
I would have sworn that the uncode value was shown with the name of the character. Now I'm wondering what editor I was thinking of. And I never use the Reports, so wouldn't have thought of looking there.

A search for the unicode value has it called "END OF GUARDED AREA" or "End of Protected Area" and there is a start character as well.
Quote:
Originally Posted by DoctorOhh View Post
Thanks for pointing me to reports. I couldn't figure out where to find its code.

I wonder why I see it in the editor section and reports, but not the preview window?
It is at least partly a font issue. The font used in the editor and viewer doesn't have it, but a different font does. And the fact that it looks like yet-another-dash probably explain why it was used here.
davidfor is offline   Reply With Quote
Old 05-01-2018, 05:00 AM   #11
BetterRed
null operator (he/him)
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 20,458
Karma: 26645808
Join Date: Mar 2012
Location: Sydney Australia
Device: none
I often scan the Character's Report to see if there's anything unexpected - like that "LATIN CAPITAL T WITH A CARON", I'm guessing it's the first letter of a real name of someone from the old Czechoslovakia or one of its successor states.

BR
BetterRed is online now   Reply With Quote
Old 05-01-2018, 07:37 AM   #12
Sarmat89
Evangelist
Sarmat89 ought to be getting tired of karma fortunes by now.Sarmat89 ought to be getting tired of karma fortunes by now.Sarmat89 ought to be getting tired of karma fortunes by now.Sarmat89 ought to be getting tired of karma fortunes by now.Sarmat89 ought to be getting tired of karma fortunes by now.Sarmat89 ought to be getting tired of karma fortunes by now.Sarmat89 ought to be getting tired of karma fortunes by now.Sarmat89 ought to be getting tired of karma fortunes by now.Sarmat89 ought to be getting tired of karma fortunes by now.Sarmat89 ought to be getting tired of karma fortunes by now.Sarmat89 ought to be getting tired of karma fortunes by now.
 
Posts: 482
Karma: 2267928
Join Date: Nov 2015
Device: none
Quote:
Originally Posted by DoctorOhh View Post
If I search for \u0097 in Regex mode it matches the character.
It is the em dash. Apparently your document is in ISO encoding, while the headers state UTF-8.
Sarmat89 is offline   Reply With Quote
Old 05-11-2018, 10:47 PM   #13
G2B
Enthusiast
G2B began at the beginning.
 
Posts: 28
Karma: 10
Join Date: Feb 2018
Device: PC / iPad
Thank you for all the comments. Very much appreciated.

This file must indeed have had a bad conversion somewhere in its past.
G2B is offline   Reply With Quote
Old 05-12-2018, 10:26 AM   #14
deback
Book E d i t o r
deback ought to be getting tired of karma fortunes by now.deback ought to be getting tired of karma fortunes by now.deback ought to be getting tired of karma fortunes by now.deback ought to be getting tired of karma fortunes by now.deback ought to be getting tired of karma fortunes by now.deback ought to be getting tired of karma fortunes by now.deback ought to be getting tired of karma fortunes by now.deback ought to be getting tired of karma fortunes by now.deback ought to be getting tired of karma fortunes by now.deback ought to be getting tired of karma fortunes by now.deback ought to be getting tired of karma fortunes by now.
 
Posts: 432
Karma: 288184
Join Date: May 2015
Device: Laptop
When I have this problem, I will highlight, copy, and paste the characters before and after the unknown character into the Find box, and then delete the characters before and after the unknown character from the Find box. There should be some kind of blank character left in the Find box. Then you can do a Find and Replace of all the unknown characters in the entire file and change them all to the EM dash (or whatever the character should be).

(I also always use Reports to find the unknown character and other characters that need to be fixed.)
deback is offline   Reply With Quote
Reply

Tags
bug, diamonds, editor

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Help! Objects look weird User7 ePub 12 01-11-2016 05:43 PM
Seriously thoughtful this is just weird! Cyclops Lounge 10 03-15-2011 01:47 AM
Something Weird Going on in 0.7.6 Lady Fitzgerald Calibre 11 07-03-2010 10:23 AM
You know what's really weird? kennyc Writers' Corner 18 03-07-2010 08:22 PM


All times are GMT -4. The time now is 06:26 PM.


MobileRead.com is a privately owned, operated and funded community.