[GUI Plugin] Import List

[ChrisMillward]

I am trying to import a columns from CLZ Book Collector using the Import List plugin.
One particular item of data (the cover price of a book) seems to give a problem.
All the data is in the form of comma separated strings, in this case "£4.50".
For some reason the £ sign is not liked and results in the entry �4.50.
'short text, like a title' or 'Plain text' both fail
There doesn't seem to be a simple Column type of 'text'

Any suggestions or Solutions would be gratefully received

[BetterRed]

Can you enter "£4.50" into the column using metadata edit? If you can then it must be a problem with Import List?

BR

[davidfor]

Originally Posted by ChrisMillward

I am trying to import a columns from CLZ Book Collector using the Import List plugin.
One particular item of data (the cover price of a book) seems to give a problem.
All the data is in the form of comma separated strings, in this case "£4.50".
For some reason the £ sign is not liked and results in the entry �4.50.
'short text, like a title' or 'Plain text' both fail
There doesn't seem to be a simple Column type of 'text'

Any suggestions or Solutions would be gratefully received

Where are you seeing this? During the import or afterward? I would have expected this to be handled properly, but, it will depend on exactly what character is used. And it could just be the font used to display the data. Again, that will depend on exactly what character is used..

Can you post or send me a sample to look at? I can check if it is the plugin or something else.

And for the record, I would be stripping the character and loading the value into a numeric column. Then putting formatting on the display of the column to show it as a price. That will work if you only have one currency.

[ChrisMillward]

To answer BetterRed, yes, I can type the text directly into the metadata. I was mistakenly assuming that the column definitions of Calibre were being used on initial import.

To answer davidfor, the character appears during the preview stage and it doesn't matter where in the import file the £ occurs. For example I inserted the character into the title column with the same results. Searching more generally for £ problems I found this.

https://www.thesitewizard.com/html-t...orrectly.shtml

It seems that the import list plugin is expecting UTF-8 encoding and it cannot be changed for CSV files. setting Calibre to UTF-8 doesn't help either.
Using notepad and saving with UTF-8 encoding however does work and gives the desired result right the way through (wherever the £ sign is). This involves an extra step using Notepad for each group of files imported, but it is a workaround.

I did consider stripping the £ sign with search and replace and importing a proper value using a Regex format but the above is a little quicker. Having the price in ASCII form is OK for me as it won't be used as a value anywhere else.

Is the Import List plugin still under development? Could the Author or anyone else modify the plugin to change the encoding for a CSV file?

[BetterRed]

@ChrisMillward - how did you create the csv? If you used Excel, then which version?

Recent versions have a (not always easy to find) option to save a CSV in UTF-8, see ==>> Excel: UTF-8 Encode . This is the easy-to-find option in the current version:



The Import List plugin has been maintained by davidfor to maintain compatibility with the calibre core (see plugin's Version History).

Moderator Notice
I'm going to move this thread into the Import List thread

BR

[DaltonST]

Job Spy is fully Unicode compatible in Python 2 and of course Python3 is all Unicode for text strings. One of its tools is: Import CSV File to Update Metadata (Advanced). It imports easily text CSV files encoded in UTF8.

Added: Attached is your example file that I opened with Notepad and did a "Save As", specifying the encoding as UTF8 instead of the ANSI that it was originally. It no longer is corrupt.

"Laboratory Techniques In Botany","Purvis, Marion Joy","£4.55"




DaltonST

[davidfor]

Originally Posted by ChrisMillward

To answer BetterRed, yes, I can type the text directly into the metadata. I was mistakenly assuming that the column definitions of Calibre were being used on initial import.

To answer davidfor, the character appears during the preview stage and it doesn't matter where in the import file the £ occurs. For example I inserted the character into the title column with the same results. Searching more generally for £ problems I found this.

https://www.thesitewizard.com/html-t...orrectly.shtml

It seems that the import list plugin is expecting UTF-8 encoding and it cannot be changed for CSV files. setting Calibre to UTF-8 doesn't help either.
Using notepad and saving with UTF-8 encoding however does work and gives the desired result right the way through (wherever the £ sign is). This involves an extra step using Notepad for each group of files imported, but it is a workaround.

I did consider stripping the £ sign with search and replace and importing a proper value using a Regex format but the above is a little quicker. Having the price in ASCII form is OK for me as it won't be used as a value anywhere else.

Is the Import List plugin still under development? Could the Author or anyone else modify the plugin to change the encoding for a CSV file?

All that is pretty much what I expected. I couldn't remember how well the plugin handled different encodings. Could you post a sample? I have been maintaining the plugin recently and will be doing the Python 3 compatibility changes soon. I'll look at this at the same time. A sample would help make sure I don't get it wrong.

[ChrisMillward]

Originally Posted by davidfor

All that is pretty much what I expected. I couldn't remember how well the plugin handled different encodings. Could you post a sample? I have been maintaining the plugin recently and will be doing the Python 3 compatibility changes soon. I'll look at this at the same time. A sample would help make sure I don't get it wrong.

Attached is a simple .TXT file consisting of "title","Author","Price"
3 columns exported as ASCII.
i.e. "Laboratory Techniques In Botany","Purvis, Marion Joy","£4.55"
You need to edit the file extension to CSV

ASCII encoding represents the £ sign as one byte A3.
UTF-8 encoding would put two bytes C2 A3

As it turns out I didn't notice that encoding options are available for the CLZ book collector export. In fact it offers None, UTF-8 and Unicode, the default being None. I never needed to consider encoding until now!

[BetterRed]

If the CLZ program can produce UTF-8 encoded CSV output, why complicate this plugin by offering encoding conversion options. Surely it's complicated enough.

BR

[davidfor]

Originally Posted by BetterRed

If the CLZ program can produce UTF-8 encoded CSV output, why complicate this plugin by offering encoding conversion options. Surely it's complicated enough.

That might be the answer, it will depend on how hard it is to do. But, it should handle this. And I'm surprised it doesn't already. Or, that it has been this long without it being raised as a problem.

@ChrisMillward: Thanks, I'll try and look at it sometime in the next week.

[BetterRed]

Originally Posted by davidfor

That might be the answer, it will depend on how hard it is to do. But, it should handle this. And I'm surprised it doesn't already. Or, that it has been this long without it being raised as a problem.

@ChrisMillward: Thanks, I'll try and look at it sometime in the next week.

I think it was raised once or twice, probably someone using an old excel, the solution was open it in notepad and save as UTF-8.

BR

[ChrisMillward]

Thanks for your help Guys

[ChrisMillward]

Since playing about with the £ sign I now find that the custom ISBN column is no longer presented in the drop down list of 'Fields'.
The column definition is there in Calibre and can be also be seen in the Tag Browser. This worked previously, am I doing something stupid.
The Import list plugin presumably builds its list of fields from those in Calibre.
All my other custom columns are visible.

[davidfor]

Originally Posted by ChrisMillward

Since playing about with the £ sign I now find that the custom ISBN column is no longer presented in the drop down list of 'Fields'.
The column definition is there in Calibre and can be also be seen in the Tag Browser. This worked previously, am I doing something stupid.
The Import list plugin presumably builds its list of fields from those in Calibre.
All my other custom columns are visible.

What sort of column is your "ISBN" column? If it is a "column built from other columns", it won't be in the list. A common definition of an ISBN column is a template to display the ISBN value from the identifiers. Is that what you have done? If that is the case, you should use the "ID:isbn" that is in the column list

The plugin should show all other custom column types. I don't think there is any that are excluded, but, I would need to check the code to be sure.

[BetterRed]

AFAIK columns built from other columns using a template have never appeared as Fields in the Import List plugin. It wouldn't make sense if they did.

@ChrisMillward: when posting an issue about a particular plugin please post it in the relevant plugin thread. There's an sticky index with links at the top of plugins sub-forum.

BR