[GUI Plugin] Import List

[capink]

Originally Posted by Rellwood

Also, there are lists with just the title, and not the title/author. I am sitting on lists containing every audible book I have but the list only has the titles and the asin (and a y/n column indicating if the book has been deleted or not), I have a list of all the books I have purchased and deleted/returned but the list ONLY has the asin, and I can't figure out how to import it without titles/authors, especially if I don't have the book in my library.

I think that these changes would be very useful.

The author field is not mandatory. While you cannot get rid of the author control when mapping columns, you can simply set the column value to zero.

Originally Posted by Rellwood

Allow lists that don't have authors/titles but do have identifiers to import. Amazon has been allowing users to download their data, but their Kindle data for the most part has just the asin identifier and not the titles/authors of the book. I am trying to find a way of importing them.

The title field is mandatory and I have no intention of changing that. If you must import lists without titles, you can simply map the title column to an arbitrary column (you can map a single csv to more than one field), and clean it after importing using the bulk metadata edit tool.

But note that whatever column you map to the title, it must contain value for each row for it to succeed. So you can map the identifier to the title field. If your csv don't have a column with values for each row, you can simply insert the same value for your title column in your spreadsheet editor.

Originally Posted by Rellwood

Allow users to configure the title/author designated columns and not just the ones that Calibre has. Many times a book in a list will have different titles/authors than just what Calibre has, and if they don't have identifiers that makse it much more difficult to match up.

I don't understand this, if they have different title/author and have no identifiers, they are not going to match anyway.

Edit: does you mean matching books using columns other than title/author/identifiers? If so, this is a lot of work and and will end up being far from perfect for variety of reasons. Beside the work it would take, one of the main worries here is false matches ruining users metadata. I am not ruling this out altogether, but not right now.

[JSWolf]

Personally, I would not want Import List to be an ASIN lookup for Amazon eBooks. My suggestion is you do the lookup and type on the missing data.

[Rellwood]

Originally Posted by capink

The author field is not mandatory. While you cannot get rid of the author control when mapping columns, you can simply set the column value to zero.



The title field is mandatory and I have no intention of changing that. If you must import lists without titles, you can simply map the title column to an arbitrary column (you can map a single csv to more than one field), and clean it after importing using the bulk metadata edit tool.

But note that whatever column you map to the title, it must contain value for each row for it to succeed. So you can map the identifier to the title field. If your csv don't have a column with values for each row, you can simply insert the same value for your title column in your spreadsheet editor.



I don't understand this, if they have different title/author and have no identifiers, they are not going to match anyway.

Edit: does you mean matching books using columns other than title/author/identifiers? If so, this is a lot of work and and will end up being far from perfect for variety of reasons. Beside the work it would take, one of the main worries here is false matches ruining users metadata. I am not ruling this out altogether, but not right now.

I have titles that are for the books I sent to my Kindle. These titles are formatted series first. When Amazon or I download the list of books, their titles show up in my 'send to Kindle" format. I have a column with this title.

Regarding the authors, it's not a big deal but the Goodreads library export .csv separates multiple authors. So if a book is by K Webster, and Ker Dukey
Calibre might have them both as the author, but the Goodreads is only going to have one of those authors. The same with anthologies. Goodreads has just the first author and then another column with every other author.

I have a column with just the Goodreads authors.

So that is a possible reason.

[ownedbycats]

Originally Posted by Rellwood

Allow lists that don't have authors/titles but do have identifiers to import. Amazon has been allowing users to download their data, but their Kindle data for the most part has just the asin identifier and not the titles/authors of the book. I am trying to find a way of importing them.

I was going to suggest pasting the ASINs into "Add from ISBN" and then doing a search & replace to change the identifier type, but when I tried it it told me it was an invalid ISBN. Oops.

Look into calibredb though. If you're able to learn how to use it I'd be surprised if it wasn't possible in it.

[compurandom]

It should be possible to have a metadata plugin download the author / title from the ID assuming that data is available from the metadata source. I have that working with the wikidata plugin now, although I hadn't intended it for that and it might have a few bugs there.

[Rellwood]

Originally Posted by ownedbycats

I was going to suggest pasting the ASINs into "Add from ISBN" and then doing a search & replace to change the identifier type, but when I tried it it told me it was an invalid ISBN. Oops.

Look into calibredb though. If you're able to learn how to use it I'd be surprised if it wasn't possible in it.

I figured out a bit of a workaround.

Just create a title/author column in the .csv you are importing and enter "unknown". If you are importing into a library and you have the column for whatever identifier you are importing, then that identifier will import. If you are importing into a library that has those books already with those identifiers then the books will import. However, only books matching the identifier will import.

I have found that creating a blank library and importing the list as books and then using the duplicate finder to match up books in the library I wanted to import the list in by identifier, and then creating a .csv for those books only helps. Anything else I would download the metadata using the identifier.

[Rellwood]

Originally Posted by compurandom

It should be possible to have a metadata plugin download the author / title from the ID assuming that data is available from the metadata source. I have that working with the wikidata plugin now, although I hadn't intended it for that and it might have a few bugs there.

Wiki data plugin?

[capink]

Version 1.5.3

• Update: Add uuid to identifiers to be used in matching, this will not be done if the user has a custom identifier named uuid.
  
  This is useful if you want to export your book metadata to edit in an external program (e.g. excel), and re-import it back to calibre. You just have to include the uuid field in the calibre export as csv dialog, and it can be used for matching.
• Update: Add option to disable tidying titles and author fields for csv import.
  
  This addresses the problem described here.
• Fix: Separate populating csv table from populating preview table. Now csv table is populated whenever csv file is changed without having to press the preview button.

If no one reports any problems with this version, I will release it to the first post in a week or so.

[capink]

Version 1.5.2

• Fix: bool column set to undefined if cell value is empty, 'undefined' or 'n/a'

Version 1.5.3

• Update: Add uuid to identifiers to be used in matching, this will not be done if the user has a custom identifier named uuid.
• Update: Add option to disable tidying titles and author fields for csv import.
• Fix: Separate populating csv table from populating preview table. Now csv table is populated whenever csv file is changed without having to press the preview button.

Version 1.5.4

• Update: (experimental, so backup first) add ability to run wizard from cmdline using calibre-debug -r "Import List"

Tip for linux users for import speed gains:

The speed of importing books with this plugin have two issues:

• The main bottleneck here is HDD read/write speed. This is the addressed in the tip below by temporarily copying metadata.db file to a ramdisk.
• The second issue is deterioration of the speed over time (also happens with ramdisk). If you import a large number of books e.g. thousands, you will notice the speed gradually decreasing. This does not happen only with the Import List plugin, but also with any other operation that add book records to calibre, you can test this by adding thousands of books using calibre's add empty book tool, or by using the copy to library feature with a large number of books.
  
  I do not know the cause of this. It is most likely a database performance issue. Maybe someone who knows better about databases can shed more light.
  
  Anyway, to address this, the import list wizard (when invoked from calibre-debug) has an option to re-initialize the database every nth books (default is 2000) to prevent the performance drop. Note that this does not apply to plugin when used from within calibre gui, this optional feature is only available if the wizard is invoked from command line. This is all experimental so caution is required when using this cmdline mode.

So, here are the steps:

1. Make a new dir in /dev/shm (ramdisk) and copy your metadata.db file to that dir. For the purpose of the tutorial, I will suppose you made a directory called MyLibrary. So the path to the newly copied metadata.db should be /dev/shm/MyLibrary/metadata.db
2. Start the plugin from calibre-debug using the following command:
   

   calibre-debug -r "Import List"

3. A dialog will pop up. First choose the location of calibre library you want to import books into, this is the original location of your library folder.
4. In the same dialog, check the option "override database path", and press the button to choose the metadata.db you just copied i.e. /dev/shm/MyLibrary/metadata.db
5. Check the option to reinitialize the database and leave the number in the spinbox as it is i.e. 2000
6. Press OK and the usual plugin wizard should start where you can import your books and witness the speed boost.
7. After you finish this process, you should copy the metadata.db file from /dev/shm/MyLibrary/metadata.db back its original location.

A few warnings regarding the above tip

• /dev/shm is a ramdisk and not a persistent storage. You should only use it by making copy of you metadata.db file just before importing. And after the import process is finished, you should move that copy immediately to another location on the HDD. Since /dev/shm is ramdisk, as soon as you restart your computer (or have a power cut), all the data living there will be lost.
• Don't under any circumstances copy or move your whole library to /dev/shm, we only COPY the metadata.db file. Why? in addition to its ethereal nature, ramdisk is as the name implies, part of your system RAM, so anything you copy to a ramdisk will subtract from the available RAM. The metadata.db is at worst a few hundred megabytes which is OK for today's RAM capacities. Still, it wouldn't hurt to check your available RAM compared to the size of the metadata.db file before copying.
• The re-initialization step is experimental, so keep testing it while always having a backup in case something goes wrong. Also note that re-initializing the database takes time that is proportionate to the size of the library. In a library containing tens of thousands of books it may take up to a minute. So re-initializing every 2000 books is a reasonable default value, take that into account if you want to modify it.

[stardot]

Can I import from .csv without importing title and author?

I already arranged it such that the list fits the titles.

Or at least if it can be identified with title/author without importing them.

[capink]

Yes and No. In the first page of the wizard you must have at least the title field (author is not mandatory). In the second page of the wizard there is a button called "update metadata for the selected books" which opens a dialog that allows you choose what fields should be updated, there you can simply un-tick the title field and it will not be updated.

Edit: In the first page of the wizard if you do not want to use the authors column, you can map it to zero.

[stardot]

Right. Thanks a lot! Very useful tool.

[asdf asdf asdf]

I think I found two bugs:
When you let Import List create empty books, every individual word in the titles gets capitalized. This makes no sense for German for example.
Also, when you import into a column that only takes certain values, it gives no error and just imports no values for this column.

Other than that, great plugin, although it would be awesome if we could have a drop-down-field instead of numbers for linking columns.

[capink]

Originally Posted by asdf asdf asdf

When you let Import List create empty books, every individual word in the titles gets capitalized. This makes no sense for German for example.

This does not happen for me. Are you sure the problem is not in the csv file you are importing from?

Originally Posted by asdf asdf asdf

Also, when you import into a column that only takes certain values, it gives no error and just imports no values for this column.

This can be added to the data validation. I am not sure this is the way to go. Currently validation only raise error for wrong data types. The original implementation in the plugin was forgiving when it comes to non fatal errors.

Originally Posted by asdf asdf asdf

Other than that, great plugin, although it would be awesome if we could have a drop-down-field instead of numbers for linking columns.

Have you tried headers based automatic mapping via "from headers" button?

[asdf asdf asdf]

Originally Posted by capink

This does not happen for me. Are you sure the problem is not in the csv file you are importing from?

Yeah, absolutely. The source titles are normal cased. I thought it was the "Tidy" option in IL, but it made no difference. Any guess whether it might be another option/plugin that's causing this? Dragging and dropping books into Calibre works correctly.

I made a few pictures. The forced capitalization happens after clearing a match and assigning an empty book to it.

Everything as expected while linking columns to headers


Still, all good here using automatic matches


Shocking revelation after removing auto matches and assigning empty books


Sorry for the weird colors, I'm using the dark theme which ImportList doesn't seem to agree with. Just have to squint harder.

Here is my csv file in case you want to take a look at it. All data inside is public and scraped from a public wiki: https://1drv.ms/u/s!AvVlLMKl6zhJh4pY...fseHg?e=ArSQoc

Originally Posted by capink

Have you tried headers based automatic mapping via "from headers" button?

Yes, and it works great. Still, would look much cleaner and you could spot errors easier. But you're right, this is a problem that can be solved with better preparation. My column headers were often random gibberish.