[GUI Plugin] Import List

[theducks]

Originally Posted by capink

Seems to be working. Can add this to the plugin.

Hold off. It fails to Change the series Name if the Author has multiples.

Someone better at this needs to help

[capink]

Here is a new beta if anyone is interested in testing. It have a couple of fixes, with one new feature:

• You can use the web tab to import many pages per list by modifying the url as in the example below:
  
  

  https://www.goodreads.com/list/show/1.Best_Books_Ever?page={1-3}

  This will download and import the first three pages of the list.

[Rellwood]

I seem to be having trouble importing lists with isbn and isbn13 ids from Goodreads. Their library export shows those columns as being ="isbn". I use excel and manually remove the =" and set it as a number. When I go in and check the isbn with quality check they all turn up as being invalid.

How can I import these columns so that they aren't invalid?

[Rellwood]

I think I figured out what causes it sometimes takes a longer time to update columns. The length of time it takes is in direct proportion to the following.

1. The columns it is updating must shouldn't be filled with something already. The more data that is is copying over the longer it takes. If you are updating a tags column, it is better to either delete all the current tags, or have a blank tags column to fill up then copy it over to the desired column. The same is true of date columns, it is better to only update cells with actual dates rather than just a blanket update all if there is data already in the columns. The thing that makes the biggest difference is when you are updating a lot of new information in a column that already has a lot of information. Like a tags column.

2. Lists that require a lot of books to be matched manually should be imported as follows
Delete any unmatched books from the list, update the metadata and import the matched books. Depending on how many unmatched books there were you can do a couple of things -

1 - rerun the list again - using different parameters for matching hoping you match what you didn't match before ie. using an id instead of title author. Only update the matched books that haven't already been matched. I keep an extra yes/no column that I add to every imported list as a "Yes" right down the line so that when I import new books I can see if they have already been imported previously. Delete any previously imported book from the list, (you can see if it has been imported because that yes/no column will let you know) delete any unmatched book, and update new matches.

2 - Import the list, delete the unmatched books, update the new matched books then chunk up the .csv into smaller amounts using a csv file chunker based on how many unmatched there were. The goal is to keep each chunk with only a small amount of unmatched. Delete all the matched books in each imported chunk (they already got matched in the first run). Match up each unmatched and update the metadata and run it. Hopefully there are not too many books to update in each chunk so it won't take as long.

For a long list with a lot of different ways to match up, I tend to do both one and two. Step one means updating the metadata for books matched up different ways through different imports of the same lsit. Then once I have imported every possible matched book, I then chunk up the list and reimport each chunk - deleting the already matched books from the list and updating the unmatched. If you do it right you will only be updating a couple of books each chunk and then it will be a lot faster than if you just tried to import the whole list.

The ultimate goal to make the importing go faster than one book for three or four seconds is to make sure the books match without needing to manually do it, and make sure the columns that are being updated with new metadata are blank or have small bits of data - and don't just mass update a column when you see that you only need to update a couple of books. The less copying over current data and the less importing columns with no data (update metadata for a whole column even if only a couple of books need updating - like a date column). It isn't so bad to copy over a yes/no or single data column than it is to update a tags column with a ton of new data.

This seems like a lot of work for the casual list importer, but if you are like me and import calabre catalogs from other libraries, the Goodreads CSV or other lists, these steps will save a lot of time in the long run. Again, this is only if you find yourself sitting around forever while a list imports - and this happens a lot.

[Twirlip]

Is there an easy way to get Amazon Kindle book purchase dates into Calibre? The purchase dates are on Amazon's "Manage Your Content and Devices" webpage.

I couldn't get Import List's web page option to work - I get "Failed to run load_page_in_browser" (someone mentioned it doesn't work on pages that require authentication). I also couldn't get the clipboard option to work - I think because when I copy from the web page, metadata from a single book is spread over multiple lines of text (see below).

I finally managed to copy from the webpage to a text file, use a Python EBNF parser (attached) to convert to CSV, then use the CSV option to import to Calibre.

Did I miss a simpler option? Or could Import List be modified to allow importing of this kind of data? I would like to go straight from the Amazon webpage into Calibre, without messing about with intermediate files.

The cut-n-paste data I'm trying to read looks like this. The author(s) and purchase date are concatenated together. The "..." in the data are really there, that's not me eliding content. The single digits are the number of collections I've put the book in (which for me is none):

...
The Consuming Fire (The Interdependency Book 2) (English Edition) 
John Scalzi18 December 2019
0
...
The Collapsing Empire (The Interdependency Book 1) (English Edition) 
READ
John Scalzi10 December 2019
0
...
Shadow Captain (Revenger) (English Edition) 
Alastair Reynolds18 October 2019
0
...
Revenger (English Edition) 
Alastair Reynolds17 October 2019
0
...
Permafrost (English Edition) 
Update Available
Alastair Reynolds12 October 2019
0

This is from amazon.de (hence the reference to English editions), in case you get different results from other countries.

Finally, thanks to whoever for a really usefull plugin

[Twirlip]

I've modifed import list for myself, so if it finds a "\n" in the regex, it uses Python's finditer to repeatedly find matches, instead of going line by line. I also updated the handling of custom fields, as it didn't work for me when using the clipboard source. I can now use this regex to parse my text:

^.*\.\.\.\n(?P<title>.*)\n(READ\n)?(Update Available\n)?(?P<authors>.*\D(?=\d?\d \w+ \d{4}))(?P<#purchased>\d?\d \w+ \d{4})\n\d+$

As a simpler example, if I have text formatted as title then author on separate lines, like this:

Pride and Prejudice
Jane Austen

To Kill a Mockingbird
Harper Lee

Then this expression works:

(?P<title>.+)\n(?P<authors>.*)\n*

If anyone wants to try my version, here it is:

[ZuleikaD]

I'm having trouble getting fields to match on an import and I'm hoping someone can help me figure out what I'm doing wrong.

Background: I'm trying to import a large number of "empty" books using data from Goodreads (ultimately to shift whole catalog management from GR to Calibre). I've created an empty library to use as a test with the same columns I have in my real library. I exported my GR data and extracted about 200 lines to test with. Saved as a csv file.

When I try to match the fields I'm not getting an option to match the ISBN field. I've tried re-naming the column in my csv file to "ID:ISBN" which is what I understand the field should be, but it's still not showing up.

(My library column layout does have an ISBN column, but since this is a column from another column, I understand that this is not the field to match to.)

I'm sure I've missed a step somewhere, but I can't figure out where!

Screenshot attached.

[ZuleikaD]

Oops, here's the screenshot:

[capink]

The plugin gets the list of IDs from the books in the library (since there can unlimited number of identifier types). If your library is empty, or if it has no books with and isbn identifier, the ID:ISBN field will not show up.

The solution to this is to create an empty book, and add dummy isbn identifier to that book. e.g. isbn:123456789. You can delete this book after you complete your import.

Edit: to create an empty book: Add Books > add an empty book.

[theducks]

Right click the Add Icon: Add from ISBN can create Empty books AND it will try and fill the metadata and covers, ( this uses you normal Metadata source configuration, so you might want to temporarily trim that)

Then you can do more with Import List

[ZuleikaD]

Originally Posted by capink

The plugin gets the list of IDs from the books in the library (since there can unlimited number of identifier types). If your library is empty, or if it has no books with and isbn identifier, the ID:ISBN field will not show up.

The solution to this is to create an empty book...

Thank you! This was the solution.

[Rellwood]

Originally Posted by ZuleikaD

I'm having trouble getting fields to match on an import and I'm hoping someone can help me figure out what I'm doing wrong.

Background: I'm trying to import a large number of "empty" books using data from Goodreads (ultimately to shift whole catalog management from GR to Calibre). I've created an empty library to use as a test with the same columns I have in my real library. I exported my GR data and extracted about 200 lines to test with. Saved as a csv file.

When I try to match the fields I'm not getting an option to match the ISBN field. I've tried re-naming the column in my csv file to "ID:ISBN" which is what I understand the field should be, but it's still not showing up.

(My library column layout does have an ISBN column, but since this is a column from another column, I understand that this is not the field to match to.)

I'm sure I've missed a step somewhere, but I can't figure out where!

Screenshot attached.

You need to have a book with an isbn number already in the library for it to show up in the plugin. I currently have all the identifiers with their own columns and each column has at least one book that has it.

I suggest creating an empty book with a popular title and downloading the metadata so that it will populate with all the identifiers you are looking for. Then you add the list from GR.

[Rellwood]

Originally Posted by Twirlip

Is there an easy way to get Amazon Kindle book purchase dates into Calibre? The purchase dates are on Amazon's "Manage Your Content and Devices" webpage.

I couldn't get Import List's web page option to work - I get "Failed to run load_page_in_browser" (someone mentioned it doesn't work on pages that require authentication). I also couldn't get the clipboard option to work - I think because when I copy from the web page, metadata from a single book is spread over multiple lines of text (see below).

I finally managed to copy from the webpage to a text file, use a Python EBNF parser (attached) to convert to CSV, then use the CSV option to import to Calibre.

Also, this website is also good for parsing the Amazon lists

https://www.ardenstone.com/2015/01/1...s-from-amazon/

Did I miss a simpler option? Or could Import List be modified to allow importing of this kind of data? I would like to go straight from the Amazon webpage into Calibre, without messing about with intermediate files.

The cut-n-paste data I'm trying to read looks like this. The author(s) and purchase date are concatenated together. The "..." in the data are really there, that's not me eliding content. The single digits are the number of collections I've put the book in (which for me is none):

...
The Consuming Fire (The Interdependency Book 2) (English Edition) 
John Scalzi18 December 2019
0
...
The Collapsing Empire (The Interdependency Book 1) (English Edition) 
READ
John Scalzi10 December 2019
0
...
Shadow Captain (Revenger) (English Edition) 
Alastair Reynolds18 October 2019
0
...
Revenger (English Edition) 
Alastair Reynolds17 October 2019
0
...
Permafrost (English Edition) 
Update Available
Alastair Reynolds12 October 2019
0

This is from amazon.de (hence the reference to English editions), in case you get different results from other countries.

Finally, thanks to whoever for a really usefull plugin

I don't know if Amazon has changed their Content and Devices page from your country, but in the US it has changed so that the 200 book list without covers where you can easily copy/paste/parse the list has changed to 20 books at a time with the full color covers- no copy/pasting or parsing is available - also it is the same with the Kindle Unlimited list. You can only go to the dedicated Kindle Unlimited page where their list is also in a non-copy/paste/parse style.

If it isn't this way for you now, then your parsing should be fine, however if they do change it there are some ways to get your information into Calibre. However it isn't as easy as it was before. Let me know if this is the case.

[sgmoore]

Has any managed to configure ImportList to parse a webpage from any of the sites that show all books for an author in reading order, for example from https://www.bookseries.org/authors/harlan-coben/ ?

Using the predefined option Fantastic Fiction, seems to give the Title but not the series.

[Twirlip]

Originally Posted by Rellwood

I don't know if Amazon has changed their Content and Devices page from your country, but in the US it has changed so that the 200 book list without covers where you can easily copy/paste/parse the list has changed to 20 books at a time with the full color covers- no copy/pasting or parsing is available - also it is the same with the Kindle Unlimited list. You can only go to the dedicated Kindle Unlimited page where their list is also in a non-copy/paste/parse style.

If it isn't this way for you now, then your parsing should be fine, however if they do change it there are some ways to get your information into Calibre. However it isn't as easy as it was before. Let me know if this is the case.

I'm in Germany, and I still get the 200 book list without covers from amazon.de. I've managed to import all my books into Calibre, so now I just have to keep it up to date. But thanks for the offer of help.