Import List plugin idea thread - Page 3

kiwidude · 07-15-2012, 06:45 PM

I made the behaviour the same as for the metadata download plugin - they don't have a series index so they are not considered part of the series. I can see that giving them a series index of zero might be a reasonable compromise though.

kiwidude · 07-15-2012, 07:14 PM

This version defaults to a series index of zero if a series name is found but no series value is specified.

Perkin · 07-18-2012, 05:41 AM

Gave the 0.4 a try, worked brilliantly on all the authors I tried it with.

Great timesaver.

Thanks kiwidude

kiwidude · 07-18-2012, 09:20 AM

Thanks Perkin, appreciate the feedback and testing as always. I will release it sometime over the next week - the only holdup is deciding just how much in the way of instructions I am going to have the chore of writing in the post...

Tom SKP · 07-18-2012, 09:21 AM

Quote:

Originally Posted by kiwidude

- to be honest I wasn't sure what direction the thread might go in. As it turned out until today it didn't really go anywhere in terms of feedback.

....

I just discovered this thread yesterday while trying to get wife to give up Excel lists containing over 2000 books.... that I have to print out for her 'little book'. And she just started a major addition to her list to add Series. Calibre would be sooooo much better and easier!!!
I've used Calibre happily for a couple of years to catalog and track my books, bit I was overwhelmed with the idea of converting wife until I saw your thread.
The only other idea I had was to import her list to Goodreads, then use your Goodreads/Calibre sync.
So this direct conversion of yours is very welcome
Thanks

Now I have to try it.

Tom

kiwidude · 07-18-2012, 03:00 PM

Hey Tom, glad you found it. I haven't had any feedback from anyone using the CSV tab as yet so I'll be interested to hear how you get on and find it all. Please post back any thoughts.

It is a 50/50 call whether in this case you might be better off doing the upload of excel to Goodreads (presuming you can get your spreadsheets in a format they will accept). The biggest challenge time-wise you are going to have is finding a goodreads id for all those books.

Fundamentally you have two choices. If you upload to Goodreads from Excel, then of course you will be using the goodreads web pages to find matches for those books. I haven't used the functionality so I don't know how well or easy that is. You might want to try it on a small sample just to test it. Once you have those books on a shelf in goodreads, then you would fire up the Goodreads Sync plugin and setup a sync from your shelf. When you do the sync, it will then give you a plugin screen showing you all those books and any matching books in calibre - if there is no match in calibre automatically made (either by goodreads id or isbn) then you have the option of creating an empty book for it. This is all the most time consuming part of the process - "linking" a book in calibre with a book in goodreads. You do it once and then never worry about it again. Once you have created those empty books, you will probably do metadata downloads for them to populate all the extra stuff you are missing like descriptions etc.

The alternative approach which is what you have cottoned onto here is to use Import List to import your books from a CSV file (exported from Excel of course). Then you will have this plugin wizard screen to match against any existing books in calibre and the option to create empty books for the rest. You might then do a metadata download to fill in all the rest of the data for them. Your next step would be to then use the Goodreads Sync plugin to add those books to shelve(s) in Goodreads. When you add to a shelf, it is going to check whether each book has a Goodreads id linked to it - if you have used the Goodreads metadata download plugin you will already have one, otherwise it will try to match by ISBN and worst case you will have to interactively use the plugins ability to search goodreads for a matching book.

Either way, for 2000 books there will be a little time involved, and it comes down to how fussy you are about which edition you want to link to on Goodreads (something you can always change later using the Goodreads Sync plugin), which matching algorithm is better based on the quality of the data in your spreadsheet, how many books you have in calibre already and whether you have been using the Goodreads metadata plugin already on them etc etc.

As I said do let me know how you get on with this plugin if you give it a whirl.

Edit: I just read the fine print on the Goodreads website. Uploading to them is only an option if you already have an ISBN for each book in your spreadsheet. Guess that answers the question as to how their matching logic works

Tom SKP · 07-25-2012, 09:21 AM

kiwidude,
Well, I didn't die off.... I have been very very busy using your Import List to set up a db for my wife's 2432 books

You were correct about Goodreads requiring an ISBN number now. That is a change from when I used it for my self about 2 years ago, so your Import List was the perfect solution at the perfect time!

Years ago she had a paper list and eventually I helped convert her to a spreadsheet, where she chose to have each author on a separate spreadsheet. She had indicator columns for books 'read' and books 'owned but not read'.
So my task was, author by author, to copy the 'read', 'owned', and 'futures' onto 3 separate CSV files and then use Import List to enter Calibre.
Your program worked flawlessly and of course picked up duplicates that entered her list, primarily due to anthologies. I would hate to think of manually entering all those 2432 books

The largest amount of time was spent on downloading the metadata. I quickly found that I could not rely on batch metadata downloading. Too many errors and no way to catch them before they over-wrote my book data. Some errors were caused by my wife's occasional misspelling, but the biggest problem was Calibre sometimes selecting the incorrect metadata from the choices available, or the completely wrong metadata, or couldn't find data. In retrospect, it seemed often to prefer metadata that contained say 2 books, rather than the single specified book.
So I spent many many hours manually downloading metadata and covers. In fact I have a couple of hundred more to edit before I'm finished.

Anyway, I appreciate the program and would not have attempted the task with out it. Many thanks (and Paypal thanks coming

)

PS. Hope you get to see some of the Olympics in person. I live in Pennsylvania, but my daughter now lives in East Molesey and is hoping to see some events, but the prices are awful

kiwidude · 07-25-2012, 10:01 AM

Hey Tom,

Thanks very much for feeding back and delighted to hear the plugin did the job for you (thanks for the donation too btw, much appreciated). Sounds like your wife was very keen indeed on keeping track of her reading - I'm sure she is going to find it all much spankier with it all no Goodreads and trackable via calibre etc now...

You did the right thing on avoiding bulk metadata download - as you found it is a very dangerous feature to use if you don't have an isbn for the book. I fully expected the actual import of the list via the plugin to be by far the easiest part of the process and pleased it did what it needed. I will push it out as a 1.0 release sometime between now and the weekend.

Re the Olympics - it is darned expensive. Also the intersection of events that I would remotely have interest in going to see and the availability of tickets to those events is pretty close to zero. I actually live about 50m away from one of the venus that does seem to still have some tickets spare but a single two hour session of boxing or weight-lifting for hundreds of pounds doesn't grab me, particularly given all the security check dramas etc. Going to the Olympic Stadium would be cool (a colleague was one of the few fortunate to get tickets for that for his family) but I shall be watching it on TV for the most part

unboggling · 08-07-2012, 05:57 PM

Quote:

Originally Posted by Perkin

I think one site I'd like to be able to import from is FantasticFiction, selecting an author and importing there whole catalogue or sub-section (series/novels) would be handy.

@kiwidude, likewise ISFDB for list of author's novels. Though from my own experience with trying various author templates for ISFDB I'd guess this site will be problematic.

Very cool plugin.

Pepin33 · 08-08-2012, 10:56 AM

I just discovered your plugin. And it's absolutely great! It'll be very usefull for my library maintenance.

Thanks for your time... and for your plugin! It's great (again)!

Pepin33 · 08-09-2012, 06:29 AM

Another question... would it possible to add fields for custom columns data at Step 1 - import from Webpage or CSV? It would be usefull to add metadata from one site to custom Calibre columns when creating new books from lists....

kiwidude · 08-09-2012, 08:25 AM

Sorry for the late replies, been distracted by doing Sigil development the last few weeks and that is taking 100% of my time...

@unboggling - I wouldn't hold high hopes for ISFDB from the briefest of looks, given the highly unstructured way they dump data out on their pages. Though you (or anyone else) has the tools with this plugin to do exactly the same experimenting as I would do - so if you figure it out I will include it in the predefined settings.

@Pepin - you seem to be rewriting your posts a lot

. You had a question previously asking for a "Select all matched" button, which I thought was a reasonable idea, if people wanted to focus only on the books they don't have on a list and delete the ones they do. So I can add that at some point.

As for custom columns, I decided against it (for now at least) a while back in this thread. One of the problems is the UI - you can see how busy the web page settings tab is already, trying to add extra xpath expression sets in there all starts to get very messy. And then there is all the data validation etc that would have to happen to allow calibre to save it, the extra columns of data on the Step 2 screen etc. I'm not saying "never" on it because it would be useful in some circumstances to get around the limitation in calibre's metadata download functionality of being unable to scrape into a custom column. However I would rather that limitation was fixed first, and then we see realistically what use cases still exist which would justify the huge effort required to add to this plugin.

I really should release this, the only holdup has been the lack of enthusiasm for trying to write the instructions on how to *fully* use it in a hopefully clear fashion. Perhaps in a few weeks when my Sigil todo list shrinks, obviously it is fully functional for people to download above, just rather hidden...

Perkin · 08-09-2012, 09:30 AM

One extra ability on the 'Step 2' page, could you automatically do the 'create entry' for any items in the list that you double-click on.

Pepin33 · 08-09-2012, 09:47 AM

Thanks for the reply, kiwidude. I edit my first message because I thought the thing I was proposing was not very clear, so I cleaned it to remake the question later. But I see you understand perfectly what I meant.

About the custom columns, I regret to hear yo're not going to add it. I know it's a huge effort for the programmer to add this. And you're right, metadata download functionality can help with this, but (for example), what if you have a specific column, and you frequent a web where the free books have a download link? If you can import the site's list, and create a list on Calibre, adding also info for download links at the specific column, you then only need to click on every book, and Calibre'll open the browser and dowload the book for you.

This is only an example about the utility of this. Another: A custom column for older books (under review on a website), to get information about the version.

Maybe people thought this are very specific examples, but I work with both.

kiwidude · 08-09-2012, 10:04 AM

@Perkin - at the moment double-click is being used to execute a search for that book. Anyone else have an opinion either way? Worst case I can always make it a configuration option, but it would be simpler to just stick to one approach. Obviously my preference is the way it is currently - there are dedicated buttons and right clicks available to create empty books. But if enough others also want the same double-click for empty book feature I can revisit it...

07-15-2012, 07:14 PM	#32
kiwidude Calibre Plugins Developer Posts: 4,637 Karma: 2162064 Join Date: Oct 2010 Location: Australia Device: Kindle Oasis	v0.4 Beta This version defaults to a series index of zero if a series name is found but no series value is specified. Last edited by kiwidude; 08-13-2012 at 05:57 AM.

07-18-2012, 03:00 PM	#36
kiwidude Calibre Plugins Developer Posts: 4,637 Karma: 2162064 Join Date: Oct 2010 Location: Australia Device: Kindle Oasis	Hey Tom, glad you found it. I haven't had any feedback from anyone using the CSV tab as yet so I'll be interested to hear how you get on and find it all. Please post back any thoughts. It is a 50/50 call whether in this case you might be better off doing the upload of excel to Goodreads (presuming you can get your spreadsheets in a format they will accept). The biggest challenge time-wise you are going to have is finding a goodreads id for all those books. Fundamentally you have two choices. If you upload to Goodreads from Excel, then of course you will be using the goodreads web pages to find matches for those books. I haven't used the functionality so I don't know how well or easy that is. You might want to try it on a small sample just to test it. Once you have those books on a shelf in goodreads, then you would fire up the Goodreads Sync plugin and setup a sync from your shelf. When you do the sync, it will then give you a plugin screen showing you all those books and any matching books in calibre - if there is no match in calibre automatically made (either by goodreads id or isbn) then you have the option of creating an empty book for it. This is all the most time consuming part of the process - "linking" a book in calibre with a book in goodreads. You do it once and then never worry about it again. Once you have created those empty books, you will probably do metadata downloads for them to populate all the extra stuff you are missing like descriptions etc. The alternative approach which is what you have cottoned onto here is to use Import List to import your books from a CSV file (exported from Excel of course). Then you will have this plugin wizard screen to match against any existing books in calibre and the option to create empty books for the rest. You might then do a metadata download to fill in all the rest of the data for them. Your next step would be to then use the Goodreads Sync plugin to add those books to shelve(s) in Goodreads. When you add to a shelf, it is going to check whether each book has a Goodreads id linked to it - if you have used the Goodreads metadata download plugin you will already have one, otherwise it will try to match by ISBN and worst case you will have to interactively use the plugins ability to search goodreads for a matching book. Either way, for 2000 books there will be a little time involved, and it comes down to how fussy you are about which edition you want to link to on Goodreads (something you can always change later using the Goodreads Sync plugin), which matching algorithm is better based on the quality of the data in your spreadsheet, how many books you have in calibre already and whether you have been using the Goodreads metadata plugin already on them etc etc. As I said do let me know how you get on with this plugin if you give it a whirl. Edit: I just read the fine print on the Goodreads website. Uploading to them is only an option if you already have an ISBN for each book in your spreadsheet. Guess that answers the question as to how their matching logic works Last edited by kiwidude; 07-18-2012 at 03:48 PM.

08-08-2012, 10:56 AM	#40
Pepin33 Zealot Posts: 109 Karma: 419 Join Date: Aug 2012 Location: Spain Device: Kindle Touch	Suggestion for the plugin I just discovered your plugin. And it's absolutely great! It'll be very usefull for my library maintenance. Thanks for your time... and for your plugin! It's great (again)! Last edited by Pepin33; 08-09-2012 at 07:54 AM.

08-09-2012, 06:29 AM	#41
Pepin33 Zealot Posts: 109 Karma: 419 Join Date: Aug 2012 Location: Spain Device: Kindle Touch	Another question... would it possible to add fields for custom columns data at Step 1 - import from Webpage or CSV? It would be usefull to add metadata from one site to custom Calibre columns when creating new books from lists.... Last edited by Pepin33; 08-09-2012 at 07:55 AM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
[Old Thread] Feature Idea - Auto convert on import	jphphotography	Calibre	6	11-04-2012 09:17 PM
[GUI Plugin] WebOS Kindle-Import	CranstD	Plugins	0	01-24-2012 03:36 PM
No Module name Tkinter on plugin import	foghat	Plugins	1	11-11-2010 07:11 PM
New Plugin Type Idea: Library Plugin	cgranade	Plugins	3	09-15-2010 12:11 PM
Run plugin before import	dremo	Plugins	6	01-09-2009 12:40 PM

07-15-2012, 06:45 PM	#31
kiwidude Calibre Plugins Developer Posts: 4,637 Karma: 2162064 Join Date: Oct 2010 Location: Australia Device: Kindle Oasis	I made the behaviour the same as for the metadata download plugin - they don't have a series index so they are not considered part of the series. I can see that giving them a series index of zero might be a reasonable compromise though.

07-18-2012, 05:41 AM	#33
Perkin Guru Posts: 655 Karma: 64171 Join Date: Sep 2010 Location: Kent, England, Sol 3, ZZ9 plural Z Alpha Device: Sony PRS-300, Kobo Aura HD, iPad (Marvin)	Gave the 0.4 a try, worked brilliantly on all the authors I tried it with. Great timesaver. Thanks kiwidude

07-18-2012, 09:20 AM	#34
kiwidude Calibre Plugins Developer Posts: 4,637 Karma: 2162064 Join Date: Oct 2010 Location: Australia Device: Kindle Oasis	Thanks Perkin, appreciate the feedback and testing as always. I will release it sometime over the next week - the only holdup is deciding just how much in the way of instructions I am going to have the chore of writing in the post...

07-25-2012, 09:21 AM	#37
Tom SKP Connoisseur Posts: 92 Karma: 1000100 Join Date: Oct 2010 Device: Samsung tab 3 & Nook Simple Touch with Glow light	kiwidude, Well, I didn't die off.... I have been very very busy using your Import List to set up a db for my wife's 2432 books You were correct about Goodreads requiring an ISBN number now. That is a change from when I used it for my self about 2 years ago, so your Import List was the perfect solution at the perfect time! Years ago she had a paper list and eventually I helped convert her to a spreadsheet, where she chose to have each author on a separate spreadsheet. She had indicator columns for books 'read' and books 'owned but not read'. So my task was, author by author, to copy the 'read', 'owned', and 'futures' onto 3 separate CSV files and then use Import List to enter Calibre. Your program worked flawlessly and of course picked up duplicates that entered her list, primarily due to anthologies. I would hate to think of manually entering all those 2432 books The largest amount of time was spent on downloading the metadata. I quickly found that I could not rely on batch metadata downloading. Too many errors and no way to catch them before they over-wrote my book data. Some errors were caused by my wife's occasional misspelling, but the biggest problem was Calibre sometimes selecting the incorrect metadata from the choices available, or the completely wrong metadata, or couldn't find data. In retrospect, it seemed often to prefer metadata that contained say 2 books, rather than the single specified book. So I spent many many hours manually downloading metadata and covers. In fact I have a couple of hundred more to edit before I'm finished. Anyway, I appreciate the program and would not have attempted the task with out it. Many thanks (and Paypal thanks coming ) PS. Hope you get to see some of the Olympics in person. I live in Pennsylvania, but my daughter now lives in East Molesey and is hoping to see some events, but the prices are awful

07-25-2012, 10:01 AM	#38
kiwidude Calibre Plugins Developer Posts: 4,637 Karma: 2162064 Join Date: Oct 2010 Location: Australia Device: Kindle Oasis	Hey Tom, Thanks very much for feeding back and delighted to hear the plugin did the job for you (thanks for the donation too btw, much appreciated). Sounds like your wife was very keen indeed on keeping track of her reading - I'm sure she is going to find it all much spankier with it all no Goodreads and trackable via calibre etc now... You did the right thing on avoiding bulk metadata download - as you found it is a very dangerous feature to use if you don't have an isbn for the book. I fully expected the actual import of the list via the plugin to be by far the easiest part of the process and pleased it did what it needed. I will push it out as a 1.0 release sometime between now and the weekend. Re the Olympics - it is darned expensive. Also the intersection of events that I would remotely have interest in going to see and the availability of tickets to those events is pretty close to zero. I actually live about 50m away from one of the venus that does seem to still have some tickets spare but a single two hour session of boxing or weight-lifting for hundreds of pounds doesn't grab me, particularly given all the security check dramas etc. Going to the Olympic Stadium would be cool (a colleague was one of the few fortunate to get tickets for that for his family) but I shall be watching it on TV for the most part

08-09-2012, 08:25 AM	#42
kiwidude Calibre Plugins Developer Posts: 4,637 Karma: 2162064 Join Date: Oct 2010 Location: Australia Device: Kindle Oasis	Sorry for the late replies, been distracted by doing Sigil development the last few weeks and that is taking 100% of my time... @unboggling - I wouldn't hold high hopes for ISFDB from the briefest of looks, given the highly unstructured way they dump data out on their pages. Though you (or anyone else) has the tools with this plugin to do exactly the same experimenting as I would do - so if you figure it out I will include it in the predefined settings. @Pepin - you seem to be rewriting your posts a lot . You had a question previously asking for a "Select all matched" button, which I thought was a reasonable idea, if people wanted to focus only on the books they don't have on a list and delete the ones they do. So I can add that at some point. As for custom columns, I decided against it (for now at least) a while back in this thread. One of the problems is the UI - you can see how busy the web page settings tab is already, trying to add extra xpath expression sets in there all starts to get very messy. And then there is all the data validation etc that would have to happen to allow calibre to save it, the extra columns of data on the Step 2 screen etc. I'm not saying "never" on it because it would be useful in some circumstances to get around the limitation in calibre's metadata download functionality of being unable to scrape into a custom column. However I would rather that limitation was fixed first, and then we see realistically what use cases still exist which would justify the huge effort required to add to this plugin. I really should release this, the only holdup has been the lack of enthusiasm for trying to write the instructions on how to fully use it in a hopefully clear fashion. Perhaps in a few weeks when my Sigil todo list shrinks, obviously it is fully functional for people to download above, just rather hidden...

08-09-2012, 09:30 AM	#43
Perkin Guru Posts: 655 Karma: 64171 Join Date: Sep 2010 Location: Kent, England, Sol 3, ZZ9 plural Z Alpha Device: Sony PRS-300, Kobo Aura HD, iPad (Marvin)	One extra ability on the 'Step 2' page, could you automatically do the 'create entry' for any items in the list that you double-click on.

08-09-2012, 09:47 AM	#44
Pepin33 Zealot Posts: 109 Karma: 419 Join Date: Aug 2012 Location: Spain Device: Kindle Touch	Thanks for the reply, kiwidude. I edit my first message because I thought the thing I was proposing was not very clear, so I cleaned it to remake the question later. But I see you understand perfectly what I meant. About the custom columns, I regret to hear yo're not going to add it. I know it's a huge effort for the programmer to add this. And you're right, metadata download functionality can help with this, but (for example), what if you have a specific column, and you frequent a web where the free books have a download link? If you can import the site's list, and create a list on Calibre, adding also info for download links at the specific column, you then only need to click on every book, and Calibre'll open the browser and dowload the book for you. This is only an example about the utility of this. Another: A custom column for older books (under review on a website), to get information about the version. Maybe people thought this are very specific examples, but I work with both.

08-09-2012, 10:04 AM	#45
kiwidude Calibre Plugins Developer Posts: 4,637 Karma: 2162064 Join Date: Oct 2010 Location: Australia Device: Kindle Oasis	@Perkin - at the moment double-click is being used to execute a search for that book. Anyone else have an opinion either way? Worst case I can always make it a configuration option, but it would be simpler to just stick to one approach. Obviously my preference is the way it is currently - there are dedicated buttons and right clicks available to create empty books. But if enough others also want the same double-click for empty book feature I can revisit it...

Advert

Advert