Special CHaracters

[Rellwood]

I know when importing a .csv (yes I know this is a plugin) that I should have it saved UTF-8 encoded.

Since the last time I updated Calibre I don't seem to be able to get those special characters to show up. I am getting the diamonds with the question marks inside.

I updated Java, and am sure that I am saving the files in UTF-8, but nothing fixes it.

Am I missing something?

Edit, am going to bed (it's 11:30pm) so I will check back on this tomorrow. If there is any response I will reply then.

Thanks!

[Sunlite]

Encoding problems, in my expirience, show up as gibberish letters. If you see the replacement char, that normally means, that the rendering engine understands what character should be displayed, but the used font does not contain it.

Maybe try to use another fully UTF-8 compatible font and see if that helps.

[DaltonST]

You use Job Spy already, so try its new "Import CSV File to Update Metadata" tool instead of the other tool you are using in order to test whether it is your .csv encoding or your screen display font or the tool you used.

"Import CSV File to Update Metadata" is fully Unicode UTF8 compliant. See the images from when it was last enhanced, which include its tooltip user guide.

BTW, I will be upgrading all 61 JS tools for Python 3 and future Calibre 3.42+ compatibility in May.


DaltonSTShow Tooltip

[Rellwood]

Originally Posted by DaltonST

You use Job Spy already, so try its new "Import CSV File to Update Metadata" tool instead of the other tool you are using in order to test whether it is your .csv encoding or your screen display font or the tool you used.

"Import CSV File to Update Metadata" is fully Unicode UTF8 compliant. See the images from when it was last enhanced, which include its tooltip user guide.

BTW, I will be upgrading all 61 JS tools for Python 3 and future Calibre 3.42+ compatibility in May.


DaltonSTShow Tooltip

Yes I use Job Spy, and I have tried to use the import csv, but it is too complicated and will not import unless each column has a defined column name.

Most of the csv files I import are broken down pieces of a single file. I use CSV chunker to break up 3500 to 5000 line files into pieces of 100 lines. Only the first page contains the headers. With the list import plugin, I just set up the columns once and each subsequent import doesn't need to be updated.

Plus, I am not sure how to import multiple columns at once. It looks like I can only import one named column at a time. Let me try it again, see if I am doing something wrong. Regardless, unless I can import files without the headers, then it won't work for my needs.

The GUI for import list is much easier than the GUI for import csv file on Job Spy.

I wish I knew how to use it better. It seems to offer a lot more than simple list importing. However, with the import list plugin I just match up columns in my library with the columns (or column number if there are no headers) with the csv and then I match up books in my library and select exactly which columns in my library to update with the data in the csv. It seems like your import list doesn't allow for that kind of choice. If I am reading the instructions right I am to match up only a single column, and then the import feature matches it to the books in the library and automatically makes the changes. I want more control over which books get changed with what metadata.

Most of my .csv files contain titles, series, series indexes, bookshelves, dates (added, read, updated, goodreads), genres, tags, etc. I need to be able to manually choose what books are being matched and manually select which columns to update.

[Rellwood]

Originally Posted by Sunlite

Encoding problems, in my expirience, show up as gibberish letters. If you see the replacement char, that normally means, that the rendering engine understands what character should be displayed, but the used font does not contain it.

Maybe try to use another fully UTF-8 compatible font and see if that helps.

Ah.. thank you, I might need to see if this works. The only characters that I am having trouble with are the accent (can't find this on my keyboard) characters for french words, and some apostrophe's it seems that only the apostrophes after certain letters are being translated as accent characters.

[DaltonST]

Originally Posted by Rellwood

Yes I use Job Spy, and I have tried to use the import csv, but it is too complicated and will not import unless each column has a defined column name.

Most of the csv files I import are broken down pieces of a single file. I use CSV chunker to break up 3500 to 5000 line files into pieces of 100 lines. Only the first page contains the headers. With the list import plugin, I just set up the columns once and each subsequent import doesn't need to be updated.

Plus, I am not sure how to import multiple columns at once. It looks like I can only import one named column at a time. Let me try it again, see if I am doing something wrong. Regardless, unless I can import files without the headers, then it won't work for my needs.

The GUI for import list is much easier than the GUI for import csv file on Job Spy.

I wish I knew how to use it better. It seems to offer a lot more than simple list importing. However, with the import list plugin I just match up columns in my library with the columns (or column number if there are no headers) with the csv and then I match up books in my library and select exactly which columns in my library to update with the data in the csv. It seems like your import list doesn't allow for that kind of choice. If I am reading the instructions right I am to match up only a single column, and then the import feature matches it to the books in the library and automatically makes the changes. I want more control over which books get changed with what metadata.

Most of my .csv files contain titles, series, series indexes, bookshelves, dates (added, read, updated, goodreads), genres, tags, etc. I need to be able to manually choose what books are being matched and manually select which columns to update.

You are mistaken about everything you said with 1 exception: each csv must have a column header for every column. Multiple headers means multiple csv files.


That tool is incredibly flexible, offers total and granular control, and uses dropdowns to make it easy. View the tooltips. View the images in the JS thread.
Show Tooltip

[Rellwood]

Originally Posted by DaltonST

You are mistaken about everything you said with 1 exception: each csv must have a column header for every column. Multiple headers means multiple csv files.


That tool is incredibly flexible, offers total and granular control, and uses dropdowns to make it easy. View the tooltips. View the images in the JS thread.
Show Tooltip

Thanks for letting me know I was wrong. I will check up on it.

However, the fact that multiple headers mean multiple .csv files and every column needing a header won't work for me in most instances.

An example would be Goodreads. Although I use the sync plugin for some of the shelves, I have way too many to set up rules and use the sync/add plugin. Plus I don't want to ruin my library if I make a rule or sync mistake and it updates everything with it. Therefore I end up using the Goodreads export library .csv file generated from the website.

(if you don't know what I am talking about, it is listed at the bottom of the bookshelf column in the "my books" page under "import/export library")

I have almost 4000 books shelved.

The export library .csv has about 8 to 10 columns I import after editing the .csv.

Because it isn't a good idea to import such a large file, I chunk it up into 100 line pieces. However, the CSV chunker program I use doesn't create new column headers for each piece.

So if I use your program, I would have to import 40 pages, 8 to 10 times each page, then go into each piece and manually create new column headers, or at best remember which bit of data would fit in which column (I can see what would be a title without having the header) and reset the program each time for each column.

I apologize, but this will not do. I will keep it in mind for other times though.

[DaltonST]

Tanjamuse uses 14,000 row csv files with the JS tool. No reason to chunk it up.

[Rellwood]

I chunk them up so that they are more manageable to import into Calibre. I am not matching up 1500 books at once (the ones that won't match up without help) and I am not updating 4000 books taking forever to update the database. I have sat and watched the .db journal file take as long as 5 seconds per book to update the database sometimes. Am not doing that with 4000 books.

:-)

I know you are trying to help me here, and I thank you so much for it! :-)

[Rellwood]

Originally Posted by DaltonST

Tanjamuse uses 14,000 row csv files with the JS tool. No reason to chunk it up.

What is Tanjamuse? I looked it up but couldn't figure it out.

[DNSB]

Originally Posted by Rellwood

What is Tanjamuse? I looked it up but couldn't figure it out.

Tanjamuse is a poster in these forums.