External Index comparison?

[MontyJ]

Hola to All!

Perhaps I have missed it, but I am looking for a plugin (or other?) that will me to compare the current/active library of books against an "external" list of books.

The reason is of course to eliminate duplicates. My collection is simply too large (80K titles) to keep within Calibre and is maintained on the web. Calibre is of course used for cleaning up titles, authors, conversion, editing, adding covers, metadata, etc., but are not saved locally.

I do maintain a separate spreadsheet of all titles, but using it to compare against new entries is cumbersome, to say the least!

I would like to have an external CSV file (or plain text, or whatever) with all my current tittles that Calibre can check against once I have cleaned up a new set of titles in the active library.

Thoughts, ideas?

Thanks!

[kovidgoyal]

use the calibre create catalog feature to generate a catalog of the calibre library in csv format. Then use your favorite spreadsheet program to compare them.

[MontyJ]

Kovid,

Thanks for the idea! That should save me an extra step that I am now doing when I get to the compare process. I was hoping to 'automate' it even further, but will see if I can use the csv catalog approach.

Again, thanks!

[MontyJ]

I just went through the process using the CSV catalog. While it does work and save me one extra step, I would still like something more 'automated'. Yes, I realize there may be nothing out there right now, but here is an idea a plugin developer might consider:

The "Find Duplicates" plugin works great for comparing titiles within a given library; and it works great. Take that idea but let it compare an external CSV list against the current library! And of course, in my case it would need to take large lists!

Here is how I have to do it now, using the CSV catalog approach suggested by Kovid-

1. When the active library title/author lists are cleaned up, I export them using the catalog featur.

2. I open my spreadsheet containing my large list of current titles I already have.

3. I have to go through a complicated ONE-AT-A-TIME comparison of the current library list vs the existing title archive list. This is extremely tedious for say a 300 title library comparison. I chose accuracy over speed on purpose, but boy, it is slow!

4. Once I weed out the duplicates, say 50 of the 300 are dupes, I have to then generate the dupe list, and then go back into Calibre and individually delete 50 titles out of the 300 in the library; again this could be 'automated' much better!

So there is the process as I have it now. It does work, but it is a whole bunch of tedious work. Hopefully others may have a similar need and a plugin similar to the "Find Duplicates" plugin can be worthwhile for a developer to come up with....I am ready to donate!

MontyJ

[Edit] Just wanted to add that my large master index has a very simple format: 2 Columns: Author and Title. Absolutely no other data! The author is a SINGLE name, not multiple. Titles are also simple, no series, no collection or volume info, etc.

[kiwidude]

@MontyJ - just use the "Import List" plugin which has the ability to import from a CSV file. Do that import into a second library, then use Find Duplicates plugin with its ability to compare libraries...

[MontyJ]

Kiwidude, Hey, thanks! Will give it a shot

[MontyJ]

Kiwidude,

Cannot get the plugin to get beyond Step 1. (If I need to take this to a different forum, let me know!)

My CSV list populates properly (2 columns, 22,000 records for this test), but the "Next" button is greyed out and will not advance to whatever "Step 2" is!

I have tried importing the CSV list with an empty active library as well as one with 72 books in it, same result.

Did I miss something?

[kiwidude]

Did you click on the Preview button to show the results on the right hand side, after setting which column number is which on the title and author settings on the left...

[MontyJ]

Yea, I have the preview listing in the contents panel, and the right hand panel shows the same contents, fully populated with the 2 columns of title/author.

Tried removing the plugin, reinstalling, even rebooting and of course restarting Calibre after reinstalling the plugin; no change...

[kiwidude]

@MontyJ - can you post a screenshot? Clicking the Preview button so that you can see some rows from the file displayed on the right side should be all that is required to enable the Next button of the wizard.

[MontyJ]

Here it is.

[kiwidude]

Very interesting. Looks like you have done everything correctly, so there must be a bug in there somewhere. Any chance you could PM or email me the file? Just zip and upload it somewhere...

[MontyJ]

kiwidude,

Here is the csv file; it was small enough to attach, and I didn't see a way to PM you with an attachment. If this somehow buggers the file, pm me an email address you want me to send the file to.

Montyj

Somehow the .csv or .7z file isn't attaching!
Ok, got a .zip to attach ok. This is the file I am importing, I just gave it a different name than in the original screen grab.

[MontyJ]

Kiwidude,

Just for a lark, I took the first 23 records from this file and made another .csv file with only those records; I am able to get to Step 2 using those 23 records!

So, it appears as tho there is a bad character/symbol somewhere in this file. I will look at my export settings from OOO Calc and see if I use quotes or some other setting if I can negate the error.

MJ

[Edit] I did an export from my ODS spreadsheet file with quotes (" ") surrounding all text in the two fields, and using a TAB as delimiter. I do have commas n the title fields, so I thought that may have been causing the hiccup somewhere further down in the .csv file. But even with the fields quoted and using a TAB as delimiter, it still hangs...

[Edit] I split the .csv file roughly in two; authors A-J, and then the second half K-Z. The first half works ok and lets me go on to Step 2, but the second half still will not. So if it is not something to do with the number of records, then it must be a bogus character in the K-Z file.

[MontyJ]

Kiwidude,

Ok, found a couple blank records in the k-z csv file. Once I got those cleaned up, all is well and I can get on to Step 2 with the entire a-z csv test file. When I checked the original ODS spreadsheet file, there are NO BLANKS. That means there is a bug in the conversion output to CSV...

A question: Your earlier post seemed to indicate I can then save this list in a separate library to "compare" against when I add 'new' titles to the active archive?

I did not see that option when I got to step 2, as it looked like it was making a compare of this csv list with what was in the active library? Have I got it backwards??

Thanks for the help!

MontyJ