how to tell the character encoding???

rheostaticsfan · 06-16-2010, 10:38 AM

I have problems with files in Calibre from time to time. Sometimes they're prc's that I "convert" to mobi to embed my new metadata. Sometimes they're epub or .pdb files which I convert to mobi.

Often the em dashes and/or the apostrophes and sometimes even the quotation marks are replaced with squares in the converted file.

After doing some digging here I gather that this may be an "input character encoding" problem and I need to put the appropriate encoding type into my preferences.

I cannot understand how I'm supposed to determine what my character encoding is? I tried cp1252 which I gather is common. That didn't help me, so I guess it's a different codec: but I have no idea how to figure out which one.

Can anyone help?

kovidgoyal · 06-16-2010, 10:55 AM

Te easiest solution is to simply use the transliterate unicode characters option which will replace these special characters with their plain ascii equivalents.

rheostaticsfan · 06-16-2010, 10:59 AM

I actually tried that...but it simply REMOVED all of the apostrophes in the document.

Starson17 · 06-16-2010, 11:04 AM

Quote:

Originally Posted by rheostaticsfan

After doing some digging here I gather that this may be an "input character encoding" problem and I need to put the appropriate encoding type into my preferences.

I cannot understand how I'm supposed to determine what my character encoding is? I tried cp1252 which I gather is common. That didn't help me, so I guess it's a different codec: but I have no idea how to figure out which one.

Can anyone help?

Now you see the problem that Calibre has - how does one determine the character encoding if it isn't specified? Calibre has tools to try to do it automatically, but they aren't perfect. You could look at the problem characters, try to figure out what they ought to be, then find an encoding that matches, then see if all the other characters look OK.

Most people don't do it that way, however. They just try reasonable options until one seems to work. Here are the ones I usually try:
cp1252
cp1251
latin1
utf-8

theducks · 06-16-2010, 11:10 AM

Quote:

Originally Posted by rheostaticsfan

I actually tried that...but it simply REMOVED all of the apostrophes in the document.

Similar thing happened on XP SP3: converting a simple TXT file

Curly Quotes (93,94) and Apostrophes(92) DELETED when converted to EPUB
Transliterate enabled. This is a simple TXT file, so there are no internals that declare charset (that was used)

rheostaticsfan · 06-16-2010, 11:12 AM

It seems that the book might have broken at the input stage.

I bought the book then ran it through ereader2html then input the html into calibre. The output of ereader2html looks fine, but when I click V in calibre it shows me a winzip folder (it imported as zip). If I open the book file in there the em dashes and apostrophes are replaced with squares.

At that point any converting I do won't help.

So is there a way to import html without losing emdashes and apostrophes?

For now I found the workaround of opening the html file in mobipocket creator and outputting a prc file.

It's adding extra steps to what is already a fairly arduous process. Is there a way to streamline?

kovidgoyal · 06-16-2010, 11:21 AM

http://calibre-ebook.com/user_manual/faq.html#id15

Starson17 · 06-16-2010, 02:29 PM

Quote:

Originally Posted by theducks

Curly Quotes (93,94) and Apostrophes(92) DELETED when converted to EPUB
Transliterate enabled. This is a simple TXT file, so there are no internals that declare charset (that was used)

I found the same thing going txt->txt. I expected it to convert Curly Double Quotes (93, 94) -> to Ordinary Double Quote (22) and Curly Single Quotes - Apostrophes (91, 92) to Ordinary Single Quote (27). Bug?

kovidgoyal · 06-16-2010, 02:40 PM

Did you specify the correct encoding for the TXT file in input encoding?

Starson17 · 06-16-2010, 02:45 PM

Quote:

Originally Posted by kovidgoyal

Did you specify the correct encoding for the TXT file in input encoding?

Nope.

I left it blank. In retrospect, it's obvious I needed to specify an encoding. Plus, I tried importing an html file with smart quotes. It kept all the smart quotes in the html, although it did convert the original file so that each byte now has an associated 00 byte.

Edit: Yes, adding the correct encoding CP1252 caused it to convert as expected.

Starson17 · 06-16-2010, 03:03 PM

Quote:

Originally Posted by rheostaticsfan

when I click V in calibre it shows me a winzip folder (it imported as zip).

That's normal - html files are grouped into a zip file to keep them together.

Quote:

If I open the book file in there the em dashes and apostrophes are replaced with squares.

At that point any converting I do won't help.

Possibly, but not certainly. I have lots of programs that show squares whenever they can't decide the encoding. To be sure, you have to use a hex editor and look. Even my hex editor has problems. If I look at the file in hex mode, there is an adjacent window that's supposed to show ASCII. It shows single smart quotes correctly, but the double smart quotes both appear as the right double quote. In text mode, it has even more problems, and this is all inside the same program!

Quote:

So is there a way to import html without losing emdashes and apostrophes?

My test of smart double and single quotes showed correct importing into Calibre. (Characters 0x91-0x94 ) What encoding/hex values for emdashes did you use? I tested 0x97 (emdash) and that worked.

theducks · 06-16-2010, 03:47 PM

Quote:

Originally Posted by Starson17

Nope.

I left it blank. In retrospect, it's obvious I needed to specify an encoding. Plus, I tried importing an html file with smart quotes. It kept all the smart quotes in the html, although it did convert the original file so that each byte now has an associated 00 byte.

Edit: Yes, adding the correct encoding CP1252 caused it to convert as expected.

Duh! Where do you set The encoding for a TXT file. I looked all over the TXT Input tab of preferences and the individual section on the convert. spaces, paragraphs... Yes. Encoding?
V .7.2

Starson17 · 06-16-2010, 04:20 PM

Quote:

Originally Posted by theducks

Duh! Where do you set The encoding for a TXT file. I looked all over the TXT Input tab of preferences and the individual section on the convert. spaces, paragraphs... Yes. Encoding?
V .7.2

Look & Feel | Input character encoding

theducks · 06-16-2010, 04:57 PM

Quote:

Originally Posted by Starson17

Look & Feel | Input character encoding

(and again)

rheostaticsfan · 06-16-2010, 07:36 PM

Quote:

Originally Posted by kovidgoyal

http://calibre-ebook.com/user_manual/faq.html#id15

I saw that. It seems to tell me to input the proper encoding. But that's how I started the thread. How do I know what the proper encoding is?

Or is there something I'm missing???

06-16-2010, 10:38 AM	#1
rheostaticsfan Zealot Posts: 107 Karma: 591 Join Date: May 2008 Device: kindle, iOS, Blackberry, Sony DPT (pdfs)	how to tell the character encoding??? I have problems with files in Calibre from time to time. Sometimes they're prc's that I "convert" to mobi to embed my new metadata. Sometimes they're epub or .pdb files which I convert to mobi. Often the em dashes and/or the apostrophes and sometimes even the quotation marks are replaced with squares in the converted file. After doing some digging here I gather that this may be an "input character encoding" problem and I need to put the appropriate encoding type into my preferences. I cannot understand how I'm supposed to determine what my character encoding is? I tried cp1252 which I gather is common. That didn't help me, so I guess it's a different codec: but I have no idea how to figure out which one. Can anyone help?

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Pdf to epub Turkish character encoding problem	blueresistance	Conversion	1	02-25-2011 05:31 PM
Encoding	prusaks	Recipes	0	09-27-2010 06:25 AM
how to add encoding?	nsg	Calibre	5	02-25-2009 09:51 PM
Character encoding in the filesystem	Jellby	Bookeen	1	03-30-2008 05:36 AM
FBReader fixes character encoding problem	jbenny	News	1	10-18-2007 10:50 PM

06-16-2010, 10:55 AM	#2
kovidgoyal creator of calibre Posts: 45,169 Karma: 27110894 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Te easiest solution is to simply use the transliterate unicode characters option which will replace these special characters with their plain ascii equivalents.

06-16-2010, 10:59 AM	#3
rheostaticsfan Zealot Posts: 107 Karma: 591 Join Date: May 2008 Device: kindle, iOS, Blackberry, Sony DPT (pdfs)	I actually tried that...but it simply REMOVED all of the apostrophes in the document.

06-16-2010, 11:12 AM	#6
rheostaticsfan Zealot Posts: 107 Karma: 591 Join Date: May 2008 Device: kindle, iOS, Blackberry, Sony DPT (pdfs)	It seems that the book might have broken at the input stage. I bought the book then ran it through ereader2html then input the html into calibre. The output of ereader2html looks fine, but when I click V in calibre it shows me a winzip folder (it imported as zip). If I open the book file in there the em dashes and apostrophes are replaced with squares. At that point any converting I do won't help. So is there a way to import html without losing emdashes and apostrophes? For now I found the workaround of opening the html file in mobipocket creator and outputting a prc file. It's adding extra steps to what is already a fairly arduous process. Is there a way to streamline?

06-16-2010, 11:21 AM	#7
kovidgoyal creator of calibre Posts: 45,169 Karma: 27110894 Join Date: Oct 2006 Location: Mumbai, India Device: Various	http://calibre-ebook.com/user_manual/faq.html#id15

06-16-2010, 02:40 PM	#9
kovidgoyal creator of calibre Posts: 45,169 Karma: 27110894 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Did you specify the correct encoding for the TXT file in input encoding?