Import List plugin idea thread

kiwidude · 06-15-2012, 04:46 PM

I first mentioned this idea on the Reading List thread, but have since decided it should be released as a standalone plugin.

The basic premise behind it is a 3-step wizard to allow users to import a list of books into calibre. Once you have configured a list source, you can match it against books in your library, creating empty books as needed. Finally the list can then be placed on a "Reading List" plugin list and/or just displayed directly in calibre.

A list could be:

Pasted from the clipboard, such as from recommendations in a forum post or copied from a simple web page. Use regular expressions to convert into title/optional author pairs, either from a predefined convenience dropdown or your own
Read from a CSV file, such as an export from a website like Goodreads, a catalog from calibre or any file source. Specify the delimiter, pick the columns for your title/author and job done.
Imported from a web page. I'm ultimately hoping to include predefined settings for a number of web sources such as various top 50/top 100 lists, Goodreads popular pages, new release pages etc from sites people suggest. Uses xpath expressions to identify the book data in the page.

There is save/import/export capability for configuration settings to allow you to minimise repeated effort and share with other plugin users.

To minimise the manual effort of reconciling book titles/authors against your library, I have implemented a fuzzy logic auto-matching mechanism, using some principles from the Find Duplicates plugin. Once you have the raw list data and move to step 2 of the wizard, the plugin performs a progression of permutation of identical, similar and slightly fuzzy matches of title and author to find a match for you. You can step through the remaining unmatched books and either directly search your library for a match, create an empty book if you don't have it or remove it from the list if it isn't of interest to you.

Here are some screenshots as it exists right now. I'm still making additions but if there is any initial reaction I would *love* to hear from you. Whether it is questions about the UI/behaviour or just suggestions for website pages to see included...

I will likely post a beta on this thread in the next few days.

Iocane · 07-01-2012, 09:49 PM

I came here today to see if there was some way of doing this exact thing! So as they say on 'teh internetz':

This! A thousand times, THIS!

I do have one question - In the forth screen shot, where it looks like you're comparing incoming/list books, is it possible to customize the columns that are used to compare?

Might help to explain why I'm asking - I would mostly be using this for anthologies - I like to have the anthology listed with an author of "Anthology" and then each story listed under its own author. I use a custom column ("Anthology" creatively enough) to note which anthology such and such an entry is from.

I would like Anthology to show up when I'm comparing list vs. existing entries.

I'm assuming that with this plugin, I'd be able to copy the TOC, pretty it up to match the needed syntax, and then voila, 40+ old school sci fi stories added to my library?

And I have to say the web page aspect is very very nice for the generation of wishlists and the like

Keep up the excellent work!

DoctorOhh · 07-02-2012, 05:47 AM

@Kiwidude Did you mean for this thread to be in the development forum?

kiwidude · 07-02-2012, 06:54 AM

@Iocane - it doesn't currently have the ability to customise choice of columns for the resolve page of the wizard, but I will add that to the list.

@Walt - to be honest I wasn't sure what direction the thread might go in. As it turned out until today it didn't really go anywhere in terms of feedback.

I primarily didn't really want to put it in the plugins forum because I thought it might end up confusing people when I have the "real" version of the plugin thread. I had intended to run it through a series of beta iterations for feedback. Given the current lack of feedback/interest I think I shall instead just continue to tweak it to my needs privately and then just throw a version out on a new plugin forum thread when I'm done. So if you or others want to move the thread elsewhere then go for it.

ElizabethN · 07-02-2012, 08:19 AM

I'm very interested, even if it never goes past beta. You do so much so I didn't want to add another request.

kiwidude · 07-02-2012, 08:45 AM

@ElizabethN - I never mind requests, after all I can always say no.

In this particular case I am very interested in any suggestions if the plugin is going to push a few buttons for people to make it more useful to them. I saw a post the other day from someone new to calibre who had their book collection stored in Access or Excel from memory - if they exported that as CSV this plugin already has the functionality to import that for them.

I am particularly interested in websites people would like to import from, because it is only through trying to setup xpath/regexes for those do I find out just how "usable" that tab is. I've already got built-in configurations defined which will take any Amazon "Listmania" list. I've also got any Goodreads "Listopia" list covered, as well as their "Most popular in xxx" type lists. For those sorts of sites you can just click the "Load" option in the plugin, drag/drop or type in your specific list url from the website and you are off and running. If you want to reuse that URL you can save it as your own settings.

I've also randomly tried a few other pages like Wikipedia Nebula Best novel winners etc and so far they all were fairly easy to setup. I'm still tweaking and tuning it to try to automatically strip the most common "cruft" out and maximise the likelihood of matches without manual effort but it is fully operational in its current state. I've held off on the beta due to no-one responding and let myself get diverted with the "Favourites Menu" plugin and also a big release of "Find Duplicates" I will put out there after the next calibre release.

Perkin · 07-02-2012, 09:33 AM

I think one site I'd like to be able to import from is FantasticFiction, selecting an author and importing there whole catalogue or sub-section (series/novels) would be handy.

This would have been even more welcome a couple of years ago when just started using calibre, now my collection is nearly sorted it's not as useful (to me) - unless I come across a great new author that I haven't read before and they have quite an extensive collection.

kiwidude · 07-02-2012, 09:54 AM

@Perkin - thanks for the suggestion, that is indeed a crackingly good one if it is practical. Unfortunately I know from writing the FF metadata plugin that their particular pages are a bit of a pain. Primarily because the underlying html of the page does not contain the book data, instead it is dynamically written out using javascript (do view source on a FF web page and you will see what I mean). I'm sure there is a workaround for this using webkit or something, Kovid will know.

I am guessing that such a rendering approach would not be desirable as a default (for speed reasons) but if I slapped another option checkbox on the screen there it may cater for it assuming it would indeed work... There is still no guarantees that the html would be structured sufficiently nicely to pull the books out of it but worth a look anyways.

kovidgoyal · 07-02-2012, 11:38 AM

@kiwidude: calibre.utils.jsbrowser

kiwidude · 07-02-2012, 11:43 AM

Thanks Kovid, I figured you would have hit this elsewhere in calibre. I know you meant calibre.web.jsbrowser.

kovidgoyal · 07-02-2012, 11:45 AM

jsbrowser isn't quite complete and because of the peculiarities of qtwebkit you should run each scrape in a worker process, but it should do the trick.

kiwidude · 07-02-2012, 12:03 PM

Can I do that in-process with ThreadedJob, or is the whole issue to do with memory leaks in which case it must be ParallelJob?

kovidgoyal · 07-02-2012, 12:07 PM

fork_job is the ticket. The problem is that QtWebKit must be run in the main thread. Of course if you are willing to block the main thread, possibly indefinitely, then you needn't fork.

kiwidude · 07-02-2012, 12:18 PM

Ran out of darts

. Thanks, that makes sense.

kiwidude · 07-02-2012, 02:44 PM

Having now got home and actually tried it I find that for Fantastic Fiction it seems they have now moved away from their dynamic javascript approach for the main body content of an author page anyways, so in this case the extra browser html trick isn't necessary (though I will still add support for this approach in case other sites need it).

A couple of simple XPath expressions and bingo, the ability to match against all books for an author in Fantastic Fiction is done

.

06-15-2012, 04:46 PM	#1
kiwidude Calibre Plugins Developer Posts: 4,635 Karma: 2162064 Join Date: Oct 2010 Location: Australia Device: Kindle Oasis	Import List plugin idea thread I first mentioned this idea on the Reading List thread, but have since decided it should be released as a standalone plugin. The basic premise behind it is a 3-step wizard to allow users to import a list of books into calibre. Once you have configured a list source, you can match it against books in your library, creating empty books as needed. Finally the list can then be placed on a "Reading List" plugin list and/or just displayed directly in calibre. A list could be: Pasted from the clipboard, such as from recommendations in a forum post or copied from a simple web page. Use regular expressions to convert into title/optional author pairs, either from a predefined convenience dropdown or your own Read from a CSV file, such as an export from a website like Goodreads, a catalog from calibre or any file source. Specify the delimiter, pick the columns for your title/author and job done. Imported from a web page. I'm ultimately hoping to include predefined settings for a number of web sources such as various top 50/top 100 lists, Goodreads popular pages, new release pages etc from sites people suggest. Uses xpath expressions to identify the book data in the page. There is save/import/export capability for configuration settings to allow you to minimise repeated effort and share with other plugin users. To minimise the manual effort of reconciling book titles/authors against your library, I have implemented a fuzzy logic auto-matching mechanism, using some principles from the Find Duplicates plugin. Once you have the raw list data and move to step 2 of the wizard, the plugin performs a progression of permutation of identical, similar and slightly fuzzy matches of title and author to find a match for you. You can step through the remaining unmatched books and either directly search your library for a match, create an empty book if you don't have it or remove it from the list if it isn't of interest to you. Here are some screenshots as it exists right now. I'm still making additions but if there is any initial reaction I would love to hear from you. Whether it is questions about the UI/behaviour or just suggestions for website pages to see included... I will likely post a beta on this thread in the next few days. Attached Thumbnails Last edited by kiwidude; 07-14-2012 at 01:08 PM. Reason: Updated screenshots

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
[Old Thread] Feature Idea - Auto convert on import	jphphotography	Calibre	6	11-04-2012 09:17 PM
[GUI Plugin] WebOS Kindle-Import	CranstD	Plugins	0	01-24-2012 03:36 PM
No Module name Tkinter on plugin import	foghat	Plugins	1	11-11-2010 07:11 PM
New Plugin Type Idea: Library Plugin	cgranade	Plugins	3	09-15-2010 12:11 PM
Run plugin before import	dremo	Plugins	6	01-09-2009 12:40 PM

07-01-2012, 09:49 PM	#2
Iocane Junior Member Posts: 8 Karma: 1162 Join Date: Mar 2011 Device: Sony PRS-600	I came here today to see if there was some way of doing this exact thing! So as they say on 'teh internetz': This! A thousand times, THIS! I do have one question - In the forth screen shot, where it looks like you're comparing incoming/list books, is it possible to customize the columns that are used to compare? Might help to explain why I'm asking - I would mostly be using this for anthologies - I like to have the anthology listed with an author of "Anthology" and then each story listed under its own author. I use a custom column ("Anthology" creatively enough) to note which anthology such and such an entry is from. I would like Anthology to show up when I'm comparing list vs. existing entries. I'm assuming that with this plugin, I'd be able to copy the TOC, pretty it up to match the needed syntax, and then voila, 40+ old school sci fi stories added to my library? And I have to say the web page aspect is very very nice for the generation of wishlists and the like Keep up the excellent work!

07-02-2012, 05:47 AM	#3
DoctorOhh US Navy, Retired Posts: 9,864 Karma: 13806776 Join Date: Feb 2009 Location: North Carolina Device: Icarus Illumina XL HD, Nexus 7	@Kiwidude Did you mean for this thread to be in the development forum?

07-02-2012, 06:54 AM	#4
kiwidude Calibre Plugins Developer Posts: 4,635 Karma: 2162064 Join Date: Oct 2010 Location: Australia Device: Kindle Oasis	@Iocane - it doesn't currently have the ability to customise choice of columns for the resolve page of the wizard, but I will add that to the list. @Walt - to be honest I wasn't sure what direction the thread might go in. As it turned out until today it didn't really go anywhere in terms of feedback. I primarily didn't really want to put it in the plugins forum because I thought it might end up confusing people when I have the "real" version of the plugin thread. I had intended to run it through a series of beta iterations for feedback. Given the current lack of feedback/interest I think I shall instead just continue to tweak it to my needs privately and then just throw a version out on a new plugin forum thread when I'm done. So if you or others want to move the thread elsewhere then go for it.

07-02-2012, 08:19 AM	#5
ElizabethN reader, ebook junkie Posts: 109 Karma: 436806 Join Date: Dec 2007 Location: western nebraska Device: droid, kindle, kobo, eslick, sony	I'm very interested, even if it never goes past beta. You do so much so I didn't want to add another request.

07-02-2012, 08:45 AM	#6
kiwidude Calibre Plugins Developer Posts: 4,635 Karma: 2162064 Join Date: Oct 2010 Location: Australia Device: Kindle Oasis	@ElizabethN - I never mind requests, after all I can always say no. In this particular case I am very interested in any suggestions if the plugin is going to push a few buttons for people to make it more useful to them. I saw a post the other day from someone new to calibre who had their book collection stored in Access or Excel from memory - if they exported that as CSV this plugin already has the functionality to import that for them. I am particularly interested in websites people would like to import from, because it is only through trying to setup xpath/regexes for those do I find out just how "usable" that tab is. I've already got built-in configurations defined which will take any Amazon "Listmania" list. I've also got any Goodreads "Listopia" list covered, as well as their "Most popular in xxx" type lists. For those sorts of sites you can just click the "Load" option in the plugin, drag/drop or type in your specific list url from the website and you are off and running. If you want to reuse that URL you can save it as your own settings. I've also randomly tried a few other pages like Wikipedia Nebula Best novel winners etc and so far they all were fairly easy to setup. I'm still tweaking and tuning it to try to automatically strip the most common "cruft" out and maximise the likelihood of matches without manual effort but it is fully operational in its current state. I've held off on the beta due to no-one responding and let myself get diverted with the "Favourites Menu" plugin and also a big release of "Find Duplicates" I will put out there after the next calibre release.

07-02-2012, 09:33 AM	#7
Perkin Guru Posts: 655 Karma: 64171 Join Date: Sep 2010 Location: Kent, England, Sol 3, ZZ9 plural Z Alpha Device: Sony PRS-300, Kobo Aura HD, iPad (Marvin)	I think one site I'd like to be able to import from is FantasticFiction, selecting an author and importing there whole catalogue or sub-section (series/novels) would be handy. This would have been even more welcome a couple of years ago when just started using calibre, now my collection is nearly sorted it's not as useful (to me) - unless I come across a great new author that I haven't read before and they have quite an extensive collection.

07-02-2012, 09:54 AM	#8
kiwidude Calibre Plugins Developer Posts: 4,635 Karma: 2162064 Join Date: Oct 2010 Location: Australia Device: Kindle Oasis	@Perkin - thanks for the suggestion, that is indeed a crackingly good one if it is practical. Unfortunately I know from writing the FF metadata plugin that their particular pages are a bit of a pain. Primarily because the underlying html of the page does not contain the book data, instead it is dynamically written out using javascript (do view source on a FF web page and you will see what I mean). I'm sure there is a workaround for this using webkit or something, Kovid will know. I am guessing that such a rendering approach would not be desirable as a default (for speed reasons) but if I slapped another option checkbox on the screen there it may cater for it assuming it would indeed work... There is still no guarantees that the html would be structured sufficiently nicely to pull the books out of it but worth a look anyways.

07-02-2012, 11:38 AM	#9
kovidgoyal creator of calibre Posts: 43,826 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	@kiwidude: calibre.utils.jsbrowser

07-02-2012, 11:43 AM	#10
kiwidude Calibre Plugins Developer Posts: 4,635 Karma: 2162064 Join Date: Oct 2010 Location: Australia Device: Kindle Oasis	Thanks Kovid, I figured you would have hit this elsewhere in calibre. I know you meant calibre.web.jsbrowser.

07-02-2012, 11:45 AM	#11
kovidgoyal creator of calibre Posts: 43,826 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	jsbrowser isn't quite complete and because of the peculiarities of qtwebkit you should run each scrape in a worker process, but it should do the trick.

07-02-2012, 12:03 PM	#12
kiwidude Calibre Plugins Developer Posts: 4,635 Karma: 2162064 Join Date: Oct 2010 Location: Australia Device: Kindle Oasis	Can I do that in-process with ThreadedJob, or is the whole issue to do with memory leaks in which case it must be ParallelJob?

07-02-2012, 12:07 PM	#13
kovidgoyal creator of calibre Posts: 43,826 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	fork_job is the ticket. The problem is that QtWebKit must be run in the main thread. Of course if you are willing to block the main thread, possibly indefinitely, then you needn't fork.

07-02-2012, 12:18 PM	#14
kiwidude Calibre Plugins Developer Posts: 4,635 Karma: 2162064 Join Date: Oct 2010 Location: Australia Device: Kindle Oasis	Ran out of darts . Thanks, that makes sense.

07-02-2012, 02:44 PM	#15
kiwidude Calibre Plugins Developer Posts: 4,635 Karma: 2162064 Join Date: Oct 2010 Location: Australia Device: Kindle Oasis	Having now got home and actually tried it I find that for Fantastic Fiction it seems they have now moved away from their dynamic javascript approach for the main body content of an author page anyways, so in this case the extra browser html trick isn't necessary (though I will still add support for this approach in case other sites need it). A couple of simple XPath expressions and bingo, the ability to match against all books for an author in Fantastic Fiction is done .