Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Development

Notices

Reply
 
Thread Tools Search this Thread
Old 06-15-2012, 04:46 PM   #1
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,228
Karma: 1334002
Join Date: Oct 2010
Location: London, UK
Device: Kindle Paperwhite 3G, iPad 3, iPad Air
Import List plugin idea thread

I first mentioned this idea on the Reading List thread, but have since decided it should be released as a standalone plugin.

The basic premise behind it is a 3-step wizard to allow users to import a list of books into calibre. Once you have configured a list source, you can match it against books in your library, creating empty books as needed. Finally the list can then be placed on a "Reading List" plugin list and/or just displayed directly in calibre.

A list could be:
  • Pasted from the clipboard, such as from recommendations in a forum post or copied from a simple web page. Use regular expressions to convert into title/optional author pairs, either from a predefined convenience dropdown or your own
  • Read from a CSV file, such as an export from a website like Goodreads, a catalog from calibre or any file source. Specify the delimiter, pick the columns for your title/author and job done.
  • Imported from a web page. I'm ultimately hoping to include predefined settings for a number of web sources such as various top 50/top 100 lists, Goodreads popular pages, new release pages etc from sites people suggest. Uses xpath expressions to identify the book data in the page.
There is save/import/export capability for configuration settings to allow you to minimise repeated effort and share with other plugin users.

To minimise the manual effort of reconciling book titles/authors against your library, I have implemented a fuzzy logic auto-matching mechanism, using some principles from the Find Duplicates plugin. Once you have the raw list data and move to step 2 of the wizard, the plugin performs a progression of permutation of identical, similar and slightly fuzzy matches of title and author to find a match for you. You can step through the remaining unmatched books and either directly search your library for a match, create an empty book if you don't have it or remove it from the list if it isn't of interest to you.

Here are some screenshots as it exists right now. I'm still making additions but if there is any initial reaction I would *love* to hear from you. Whether it is questions about the UI/behaviour or just suggestions for website pages to see included...

I will likely post a beta on this thread in the next few days.
Attached Thumbnails
Click image for larger version

Name:	Page1ImportClipboard.png
Views:	126
Size:	52.8 KB
ID:	89141   Click image for larger version

Name:	Page1ImportCSV.png
Views:	114
Size:	53.1 KB
ID:	89142   Click image for larger version

Name:	Page1ImportWeb.png
Views:	106
Size:	65.0 KB
ID:	89143   Click image for larger version

Name:	Page1PredefinedList.png
Views:	102
Size:	75.1 KB
ID:	89144   Click image for larger version

Name:	Page1PredefinedTree.png
Views:	106
Size:	58.6 KB
ID:	89145   Click image for larger version

Name:	Page1UserSettings.png
Views:	99
Size:	47.8 KB
ID:	89146   Click image for larger version

Name:	Page2Matches.png
Views:	105
Size:	55.7 KB
ID:	89147   Click image for larger version

Name:	Page3Save.png
Views:	93
Size:	30.7 KB
ID:	89148  

Last edited by kiwidude; 07-14-2012 at 01:08 PM. Reason: Updated screenshots
kiwidude is offline   Reply With Quote
Old 07-01-2012, 09:49 PM   #2
Iocane
Junior Member
Iocane can extract oil from cheeseIocane can extract oil from cheeseIocane can extract oil from cheeseIocane can extract oil from cheeseIocane can extract oil from cheeseIocane can extract oil from cheeseIocane can extract oil from cheeseIocane can extract oil from cheeseIocane can extract oil from cheese
 
Posts: 8
Karma: 1162
Join Date: Mar 2011
Device: Sony PRS-600
I came here today to see if there was some way of doing this exact thing! So as they say on 'teh internetz':

This! A thousand times, THIS!

I do have one question - In the forth screen shot, where it looks like you're comparing incoming/list books, is it possible to customize the columns that are used to compare?

Might help to explain why I'm asking - I would mostly be using this for anthologies - I like to have the anthology listed with an author of "Anthology" and then each story listed under its own author. I use a custom column ("Anthology" creatively enough) to note which anthology such and such an entry is from.

I would like Anthology to show up when I'm comparing list vs. existing entries.

I'm assuming that with this plugin, I'd be able to copy the TOC, pretty it up to match the needed syntax, and then voila, 40+ old school sci fi stories added to my library?

And I have to say the web page aspect is very very nice for the generation of wishlists and the like

Keep up the excellent work!
Iocane is offline   Reply With Quote
Old 07-02-2012, 05:47 AM   #3
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 8,861
Karma: 12755553
Join Date: Feb 2009
Location: North Carolina
Device: Nexus 7
@Kiwidude Did you mean for this thread to be in the development forum?
DoctorOhh is offline   Reply With Quote
Old 07-02-2012, 06:54 AM   #4
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,228
Karma: 1334002
Join Date: Oct 2010
Location: London, UK
Device: Kindle Paperwhite 3G, iPad 3, iPad Air
@Iocane - it doesn't currently have the ability to customise choice of columns for the resolve page of the wizard, but I will add that to the list.

@Walt - to be honest I wasn't sure what direction the thread might go in. As it turned out until today it didn't really go anywhere in terms of feedback.

I primarily didn't really want to put it in the plugins forum because I thought it might end up confusing people when I have the "real" version of the plugin thread. I had intended to run it through a series of beta iterations for feedback. Given the current lack of feedback/interest I think I shall instead just continue to tweak it to my needs privately and then just throw a version out on a new plugin forum thread when I'm done. So if you or others want to move the thread elsewhere then go for it.
kiwidude is offline   Reply With Quote
Old 07-02-2012, 08:19 AM   #5
ElizabethN
reader, ebook junkie
ElizabethN ought to be getting tired of karma fortunes by now.ElizabethN ought to be getting tired of karma fortunes by now.ElizabethN ought to be getting tired of karma fortunes by now.ElizabethN ought to be getting tired of karma fortunes by now.ElizabethN ought to be getting tired of karma fortunes by now.ElizabethN ought to be getting tired of karma fortunes by now.ElizabethN ought to be getting tired of karma fortunes by now.ElizabethN ought to be getting tired of karma fortunes by now.ElizabethN ought to be getting tired of karma fortunes by now.ElizabethN ought to be getting tired of karma fortunes by now.ElizabethN ought to be getting tired of karma fortunes by now.
 
ElizabethN's Avatar
 
Posts: 104
Karma: 436806
Join Date: Dec 2007
Location: western nebraska
Device: droid, kindle, kobo, eslick, sony
I'm very interested, even if it never goes past beta. You do so much so I didn't want to add another request.
ElizabethN is offline   Reply With Quote
Old 07-02-2012, 08:45 AM   #6
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,228
Karma: 1334002
Join Date: Oct 2010
Location: London, UK
Device: Kindle Paperwhite 3G, iPad 3, iPad Air
@ElizabethN - I never mind requests, after all I can always say no. In this particular case I am very interested in any suggestions if the plugin is going to push a few buttons for people to make it more useful to them. I saw a post the other day from someone new to calibre who had their book collection stored in Access or Excel from memory - if they exported that as CSV this plugin already has the functionality to import that for them.

I am particularly interested in websites people would like to import from, because it is only through trying to setup xpath/regexes for those do I find out just how "usable" that tab is. I've already got built-in configurations defined which will take any Amazon "Listmania" list. I've also got any Goodreads "Listopia" list covered, as well as their "Most popular in xxx" type lists. For those sorts of sites you can just click the "Load" option in the plugin, drag/drop or type in your specific list url from the website and you are off and running. If you want to reuse that URL you can save it as your own settings.

I've also randomly tried a few other pages like Wikipedia Nebula Best novel winners etc and so far they all were fairly easy to setup. I'm still tweaking and tuning it to try to automatically strip the most common "cruft" out and maximise the likelihood of matches without manual effort but it is fully operational in its current state. I've held off on the beta due to no-one responding and let myself get diverted with the "Favourites Menu" plugin and also a big release of "Find Duplicates" I will put out there after the next calibre release.
kiwidude is offline   Reply With Quote
Old 07-02-2012, 09:33 AM   #7
Perkin
Guru
Perkin calls his or her ebook reader Vera.Perkin calls his or her ebook reader Vera.Perkin calls his or her ebook reader Vera.Perkin calls his or her ebook reader Vera.Perkin calls his or her ebook reader Vera.Perkin calls his or her ebook reader Vera.Perkin calls his or her ebook reader Vera.Perkin calls his or her ebook reader Vera.Perkin calls his or her ebook reader Vera.Perkin calls his or her ebook reader Vera.Perkin calls his or her ebook reader Vera.
 
Perkin's Avatar
 
Posts: 644
Karma: 64171
Join Date: Sep 2010
Location: Kent, England, Sol 3, ZZ9 plural Z Alpha
Device: Sony PRS-300, Kobo Aura HD
I think one site I'd like to be able to import from is FantasticFiction, selecting an author and importing there whole catalogue or sub-section (series/novels) would be handy.

This would have been even more welcome a couple of years ago when just started using calibre, now my collection is nearly sorted it's not as useful (to me) - unless I come across a great new author that I haven't read before and they have quite an extensive collection.
Perkin is offline   Reply With Quote
Old 07-02-2012, 09:54 AM   #8
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,228
Karma: 1334002
Join Date: Oct 2010
Location: London, UK
Device: Kindle Paperwhite 3G, iPad 3, iPad Air
@Perkin - thanks for the suggestion, that is indeed a crackingly good one if it is practical. Unfortunately I know from writing the FF metadata plugin that their particular pages are a bit of a pain. Primarily because the underlying html of the page does not contain the book data, instead it is dynamically written out using javascript (do view source on a FF web page and you will see what I mean). I'm sure there is a workaround for this using webkit or something, Kovid will know.

I am guessing that such a rendering approach would not be desirable as a default (for speed reasons) but if I slapped another option checkbox on the screen there it may cater for it assuming it would indeed work... There is still no guarantees that the html would be structured sufficiently nicely to pull the books out of it but worth a look anyways.
kiwidude is offline   Reply With Quote
Old 07-02-2012, 11:38 AM   #9
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 26,121
Karma: 5101571
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
@kiwidude: calibre.utils.jsbrowser
kovidgoyal is offline   Reply With Quote
Old 07-02-2012, 11:43 AM   #10
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,228
Karma: 1334002
Join Date: Oct 2010
Location: London, UK
Device: Kindle Paperwhite 3G, iPad 3, iPad Air
Thanks Kovid, I figured you would have hit this elsewhere in calibre. I know you meant calibre.web.jsbrowser.
kiwidude is offline   Reply With Quote
Old 07-02-2012, 11:45 AM   #11
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 26,121
Karma: 5101571
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
jsbrowser isn't quite complete and because of the peculiarities of qtwebkit you should run each scrape in a worker process, but it should do the trick.
kovidgoyal is offline   Reply With Quote
Old 07-02-2012, 12:03 PM   #12
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,228
Karma: 1334002
Join Date: Oct 2010
Location: London, UK
Device: Kindle Paperwhite 3G, iPad 3, iPad Air
Can I do that in-process with ThreadedJob, or is the whole issue to do with memory leaks in which case it must be ParallelJob?
kiwidude is offline   Reply With Quote
Old 07-02-2012, 12:07 PM   #13
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 26,121
Karma: 5101571
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
fork_job is the ticket. The problem is that QtWebKit must be run in the main thread. Of course if you are willing to block the main thread, possibly indefinitely, then you needn't fork.
kovidgoyal is offline   Reply With Quote
Old 07-02-2012, 12:18 PM   #14
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,228
Karma: 1334002
Join Date: Oct 2010
Location: London, UK
Device: Kindle Paperwhite 3G, iPad 3, iPad Air
Ran out of darts . Thanks, that makes sense.
kiwidude is offline   Reply With Quote
Old 07-02-2012, 02:44 PM   #15
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,228
Karma: 1334002
Join Date: Oct 2010
Location: London, UK
Device: Kindle Paperwhite 3G, iPad 3, iPad Air
Having now got home and actually tried it I find that for Fantastic Fiction it seems they have now moved away from their dynamic javascript approach for the main body content of an author page anyways, so in this case the extra browser html trick isn't necessary (though I will still add support for this approach in case other sites need it).

A couple of simple XPath expressions and bingo, the ability to match against all books for an author in Fantastic Fiction is done .
kiwidude is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
[Old Thread] Feature Idea - Auto convert on import jphphotography Calibre 6 11-04-2012 09:17 PM
[GUI Plugin] WebOS Kindle-Import CranstD Plugins 0 01-24-2012 03:36 PM
No Module name Tkinter on plugin import foghat Plugins 1 11-11-2010 07:11 PM
New Plugin Type Idea: Library Plugin cgranade Plugins 3 09-15-2010 12:11 PM
Run plugin before import dremo Plugins 6 01-09-2009 12:40 PM


All times are GMT -4. The time now is 08:01 AM.


MobileRead.com is a privately owned, operated and funded community.