Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Plugins

Notices

Reply
 
Thread Tools Search this Thread
Old 10-01-2013, 03:47 PM   #601
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,600
Karma: 2092290
Join Date: Oct 2010
Location: Australia
Device: Kindle Oasis
@Terisa - not quite sure what "owned" information you are trying to get? This plugin is not a general metadata scraping type of plugin, so chances are it does not currently have the ability you want right now. You can trigger actions like setting values in custom columns based on a book being on a shelf, but there is only a few attributes of the book (such as last read, review text and rating) that the plugin allows you to consume into a custom column.

@arbje - as you can see at the top of every class file all the code is under the GPL v3 license, the same as calibre itself is. So by all means if you have the time to look into it then do so. As I mentioned above unfortunately this is a fairly complicated plugin mostly due to the way it "evolved" over time, so it isn't the easiest plugin to maintain.

I can't see Goodreads changing their call to bring back more review text in the list of all books API call. I am sure they did it the way they did to prevent timeouts and provide performance/reduced bandwidth for people just wanting to get the list of books on a shelf. They also seem to be very inactive when it comes to evolving the API, so any design would be best made assuming things will not change on their side.

Elements of what I propose above are not technically difficult. Plugins like Find Duplicates already use a special ability of calibre to store hidden data associated with each book, the same approach could be used to store a last synced/updated date. Regardless of the review text issue that change alone would make the plugin a lot nicer to use for people using the Sync function. Potentially the same idea should be applied to the Upload Tags as Shelves and Download Shelves as Tags features too. Though you would likely also need to have some way for the user to force a sync (i.e. ignore any stored last date in the hidden data) as they have no other way of clearing it, and if for instance they wiped their Tag data and then wanted to re-download from their Goodread shelves they would be a bit scuppered without a Force option.

There is no need to do the additional call to bring back the review text unless of course the user has chosen to sync that column back, and it only needs to take place when they click the Sync button. However what to do in regards to the long running operation is a more difficult issue. Ordinarily the pattern in calibre is to put long running tasks (as this should become, with using a timer to only query Goodreads once every few seconds) as a background task (like Count Pages, Extract ISBN etc). This allows the user to continue to use calibre while this stuff churns along in the background. Then of course you need to wire in the dialog that everyone hates but is sadly necessary to get them to confirm they are ok updating the data, at which point you are back on the UI thread and with a progress dialog block user input while it actually updates the database. This also is something I would like to see happen with the Upload/Download tags features - they should all run in the background of calibre, and all be throttled to run a lot slower than they currently do. With that in place it would be safe to remove the 50 books at a time constraint, and people could leave it running overnight if they wanted to update books in bulk at a very slow rate.

These are non-trivial changes and a *lot* of work (as in a number of days) to complete, though as I say the examples are there either within calibre itself or other plugins I have written. I don't have anywhere near that sort of time at the moment. If I had that sort of time I would be tempted just to rewrite the plugin all over again - there are parts that can be reused like all the OAUTH/web scraping stuff, but I would come up with a completely different UI. I think it is over-complicated in its current state. This was because (a) so many things got bolted on along the way, (b) the awful Goodreads T&C constraints and very limited API options constrain what is easily possible without incurring their displeasure and (c) I was arguably too willing to add features which perhaps in reality only a very small % of people actually use and I should have just said "no" for this plugin. There are also features like Terisa suggested above which come up - if the xml has data about a book, it would be nice as part of a sync or fetch operation to be able to pull this back and put it into custom columns etc, since calibre *still* does not support this in metadata download plugins...

Anyways... it is what it is. In a month or two perhaps I may feel more motivated but right now it does everything I personally need from it (and a lot that I don't) so other than making suggestions to someone else willing to do the bulk of the work it isn't likely to get too much attention in the short term from myself...
kiwidude is offline   Reply With Quote
Old 10-01-2013, 03:53 PM   #602
Terisa de morgan
Grand Sorcerer
Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.Terisa de morgan ought to be getting tired of karma fortunes by now.
 
Terisa de morgan's Avatar
 
Posts: 6,211
Karma: 11766195
Join Date: Jun 2009
Location: Madrid, Spain
Device: Kobo Clara/Aura One/Forma,XiaoMI 5, iPad, Huawei MediaPad, YotaPhone 2
Quote:
Originally Posted by kiwidude View Post
@Terisa - not quite sure what "owned" information you are trying to get? This plugin is not a general metadata scraping type of plugin, so chances are it does not currently have the ability you want right now. You can trigger actions like setting values in custom columns based on a book being on a shelf, but there is only a few attributes of the book (such as last read, review text and rating) that the plugin allows you to consume into a custom column.
I was talking about "Purchase date". There's an API function for it. I can try to add it, the problem is I don't know how to get the xml info, so I can parse it. Well, I'll try.
Terisa de morgan is offline   Reply With Quote
Old 10-01-2013, 04:52 PM   #603
hakan42
Zealot
hakan42 is on a distinguished road
 
hakan42's Avatar
 
Posts: 136
Karma: 60
Join Date: Jul 2009
Location: Munich, Germany
Device: Nook Classic rooted; Galaxy S IV with Aldiko, other older devices
Quote:
Originally Posted by kiwidude View Post
@arbje - as you can see at the top of every class file all the code is under the GPL v3 license, the same as calibre itself is. So by all means if you have the time to look into it then do so. As I mentioned above unfortunately this is a fairly complicated plugin mostly due to the way it "evolved" over time, so it isn't the easiest plugin to maintain.
@kiwidude do you already have the plugin source in a source management system somewhere? If not, would you mind putting it to a github project (where the core of calibre already is)?

This would make contributing to your plugins tremendously easier, especially with you being in a shortage of available time :-)
hakan42 is offline   Reply With Quote
Old 10-03-2013, 03:40 AM   #604
arbje
The Librarian
arbje began at the beginning.
 
Posts: 4
Karma: 10
Join Date: Sep 2013
Location: Belgium
Device: Nexus 7 (1st gen)
Quote:
Originally Posted by hakan42 View Post
@kiwidude do you already have the plugin source in a source management system somewhere? If not, would you mind putting it to a github project (where the core of calibre already is)?

This would make contributing to your plugins tremendously easier, especially with you being in a shortage of available time :-)
That ^ was what I was getting at Github preferably
arbje is offline   Reply With Quote
Old 10-24-2013, 01:41 AM   #605
macnab69
Zealot
macnab69 shares his or her toysmacnab69 shares his or her toysmacnab69 shares his or her toysmacnab69 shares his or her toysmacnab69 shares his or her toysmacnab69 shares his or her toysmacnab69 shares his or her toysmacnab69 shares his or her toysmacnab69 shares his or her toysmacnab69 shares his or her toysmacnab69 shares his or her toys
 
Posts: 129
Karma: 5754
Join Date: Jan 2012
Location: South Africa
Device: Kindle 4
I don't want to destroy my library so I don't want to experiment too much.
Assuming a new calibre user. Assume not ebook user. They have a number of shelves at Goodreads.

What settings/actions are needed to import the contents of their shelves. The Tags can be set according to shelf names.
macnab69 is offline   Reply With Quote
Old 10-24-2013, 01:42 AM   #606
macnab69
Zealot
macnab69 shares his or her toysmacnab69 shares his or her toysmacnab69 shares his or her toysmacnab69 shares his or her toysmacnab69 shares his or her toysmacnab69 shares his or her toysmacnab69 shares his or her toysmacnab69 shares his or her toysmacnab69 shares his or her toysmacnab69 shares his or her toysmacnab69 shares his or her toys
 
Posts: 129
Karma: 5754
Join Date: Jan 2012
Location: South Africa
Device: Kindle 4
Another question related to the previous question.

Is there any way we could import from Recommendations, short of copying/moving all the books to a shelf?
macnab69 is offline   Reply With Quote
Old 10-26-2013, 07:42 AM   #607
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,600
Karma: 2092290
Join Date: Oct 2010
Location: Australia
Device: Kindle Oasis
Quote:
Originally Posted by macnab69 View Post
I don't want to destroy my library so I don't want to experiment too much.
Assuming a new calibre user. Assume not ebook user. They have a number of shelves at Goodreads.

What settings/actions are needed to import the contents of their shelves. The Tags can be set according to shelf names.
Read the help file that comes with the plugin. The fundamental issue you have is that a book in calibre has to be "linked" to the matching edition in Goodreads in order to pull down any information about it. There are a variety of ways you can link a book (discussed in the help). However for the purposes of a new user (assuming they may not even have entries for the books in calibre) your best bet is to setup to Sync from those shelves from goodreads, and as part of the Sync have a rule to set a tag to match the shelf name.

There is another feature in the plugin called "Download shelves as tags", however that presupposes you have the matching books already created and linked in calibre. If instead you use the Sync feature you can do that as part of the Sync operation.
Quote:
Originally Posted by macnab69 View Post
Another question related to the previous question.

Is there any way we could import from Recommendations, short of copying/moving all the books to a shelf?
You could try the Import List plugin.
kiwidude is offline   Reply With Quote
Old 10-26-2013, 08:00 AM   #608
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,600
Karma: 2092290
Join Date: Oct 2010
Location: Australia
Device: Kindle Oasis
Quote:
Originally Posted by hakan42 View Post
@kiwidude do you already have the plugin source in a source management system somewhere? If not, would you mind putting it to a github project (where the core of calibre already is)?

This would make contributing to your plugins tremendously easier, especially with you being in a shortage of available time :-)
There are a variety of problems with this. The first and foremost is the amount of time it would take me to work out how to use github sufficently well to upload them all and establish my routine for maintaining them. I don't use git in my day job, and I know from the work I did on Sigil a while ago it is not particularly intuitive to use - powerful/flexible - yes, user friendly - definitely not.

Secondly I then have to change how all my various batch files for testing plugins work, which are all currently setup to work with some specific paths where I keep the source on my drives. Not insurmountable, but more time.

Thirdly one of the advantages of my current approach is that I can locate my plugins within the calibre_plugins folder within the calibre source tree on my drive. Which lets Eclipse work quite nicely as far as its own validation and hyperlinking for code navigation etc. I don't know if that would still be possible or problematic to have plugins from another git source stuck within Kovid's git pull. I suspect it would be a problem. Maybe eclipse can be taught to cope with a parallel directory to still get the hyperlinks directly, but again it isn't something i use in my day job so is more hours of fiddling and frustration to experiment with.

Fourthly I have a "common_utils" file shared by all my plugins. I don't know how well that might play being outside of the plugin folder itself under a git scenario, in terms of pulling updates etc. Again maybe not an issue at all, but something to consider.

Fifthly even presupposing I spend several days getting this is all setup, there is no guarantee anyone else will actually bother making contributions. As that now introduces the same burden of learning git on any other developer wanting to contribute. Whereas right now they have the zip code of source right there, along with my email address in every code file to just send me a patch or via the threads here.

And finally even if there are indeed other developers out there who allegedly would contribute but only if the plugins were in git (and I remain sceptical of that claim as we are down to a minority of a minority of a minority...) I still would have to review changes to be sure people actually knew what the heck they were doing and not introducing new problems. Unless of course it was someone who knows far more than me like Kovid/chaley etc. So that still means more of my time...

I don't deny there would be some advantages. But I don't think that it is a game breaker or a roadblock preventing others from contributing. You already have the latest version of the source code in the zip file in every thread. If someone really feels strongly about contributing they can work with that. If someone feels strongly enough about a plugin to want changes to be made and they have the skills to make the changes, not being in git is not the problem!
kiwidude is offline   Reply With Quote
Old 10-26-2013, 10:03 AM   #609
hakan42
Zealot
hakan42 is on a distinguished road
 
hakan42's Avatar
 
Posts: 136
Karma: 60
Join Date: Jul 2009
Location: Munich, Germany
Device: Nook Classic rooted; Galaxy S IV with Aldiko, other older devices
Quote:
Originally Posted by kiwidude View Post
There are a variety of problems with this. The first and foremost is the amount of time it would take me to work out how to use github sufficently well to upload them all and establish my routine for maintaining them. I don't use git in my day job, and I know from the work I did on Sigil a while ago it is not particularly intuitive to use - powerful/flexible - yes, user friendly - definitely not.
O.k., admittedly, I use source control religiously in my line of work, so I did not even think it possible that you would not keep some sort of source control on your machine. And then, given that the plugins are de-facto open source, it would have been nice to expose this to the world.

Usually, I tend to work (private, open source and paid-for work) on two dozen things at once, so keeping all my thoughts (and how I arrived at that idea) in a source control system, by now preferably git, is a lifesaver

My point would have been that with the amazing set of plugins you have written, new plugin writers would have benefited from seeing how you work. On the other hand, if something breaks unexpectedly because, say, goodreads choose to change something, and you are very busy with something else, other people would have a chance to jump in and help out.

Aaaaand, on the gripping hand, even if you lose interest in maintaining calibre plugins at some time (which I see happening all the time with Jenkins plugins), you could pass the torch to someone else.

My points all being made, would you mind if I took at least the released versions of your scripts and created a repository from them? It is always very interesting to see how other people solved certain problems
hakan42 is offline   Reply With Quote
Old 10-26-2013, 02:54 PM   #610
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,600
Karma: 2092290
Join Date: Oct 2010
Location: Australia
Device: Kindle Oasis
@hakan - if you want to create your own personal repository that only you can view and is not visible in google etc (such as by being local), of course by all means go ahead. However I would *not* be happy with someone other than me putting them up in a public source code repository. It will just create too much confusion from them being unofficial, unsupported and inevitably out of sync, particularly when I eventually do get around to putting them somewhere myelf. It will happen someday, but not right now with me working 100 hour weeks in my day job.

And what most people don't realise is that the subset of calibre users of the relevant plugin with the right technical skills, motivation and time is in all likelihood less than a handful in the whole world. If someone really cares enough to make a change, as I said above it makes not one iota of difference in my opinion about them not being on github and them opening the zip file directly instead. So I really don't buy the "people will come out of the woodwork to support the development" argument at all...
kiwidude is offline   Reply With Quote
Old 10-26-2013, 09:13 PM   #611
davidfor
Grand Sorcerer
davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.
 
Posts: 24,908
Karma: 47303748
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
Quote:
Originally Posted by hakan42 View Post
My point would have been that with the amazing set of plugins you have written, new plugin writers would have benefited from seeing how you work
We already are. At least a third of the code in my plugin is from one or other of kiwidudes plugins. And looking at plugins from others, I keep having deja vu moments. The plugins are zipped collection of python code, they are easy to open and read.
davidfor is offline   Reply With Quote
Old 10-28-2013, 08:22 PM   #612
EdStout
Member
EdStout began at the beginning.
 
Posts: 11
Karma: 10
Join Date: Jun 2013
Device: Mac, iPad
Before, when adding a book to a GoodReads shelf, I could drag a link for the desired book and edition and drop it on the Search For GoodReads Book dialog. Now I can't -- I just get the not-allowed cursor icon and nothing happens. Any ideas?

Oh yeah, I updated to Mavericks recently -- could that have anything to do with it?
EdStout is offline   Reply With Quote
Old 10-29-2013, 12:23 AM   #613
hakan42
Zealot
hakan42 is on a distinguished road
 
hakan42's Avatar
 
Posts: 136
Karma: 60
Join Date: Jul 2009
Location: Munich, Germany
Device: Nook Classic rooted; Galaxy S IV with Aldiko, other older devices
I have the same issue on Windows with calibre 1.5 and the most current version of the plugin.
hakan42 is offline   Reply With Quote
Old 11-03-2013, 09:10 AM   #614
Krazykiwi
Zealot
Krazykiwi ought to be getting tired of karma fortunes by now.Krazykiwi ought to be getting tired of karma fortunes by now.Krazykiwi ought to be getting tired of karma fortunes by now.Krazykiwi ought to be getting tired of karma fortunes by now.Krazykiwi ought to be getting tired of karma fortunes by now.Krazykiwi ought to be getting tired of karma fortunes by now.Krazykiwi ought to be getting tired of karma fortunes by now.Krazykiwi ought to be getting tired of karma fortunes by now.Krazykiwi ought to be getting tired of karma fortunes by now.Krazykiwi ought to be getting tired of karma fortunes by now.Krazykiwi ought to be getting tired of karma fortunes by now.
 
Posts: 137
Karma: 2156958
Join Date: Jan 2013
Device: Too many random androids to list
Quote:
Originally Posted by EdStout View Post
Before, when adding a book to a GoodReads shelf, I could drag a link for the desired book and edition and drop it on the Search For GoodReads Book dialog. Now I can't -- I just get the not-allowed cursor icon and nothing happens. Any ideas?

Oh yeah, I updated to Mavericks recently -- could that have anything to do with it?
GR is enforcing https, and I'm pretty sure the drag and drop stopped working same day that started. Possibly unrelated, but worth mentioning.

You can copy the url and paste it though, it works fine.
Krazykiwi is offline   Reply With Quote
Old 11-03-2013, 10:09 AM   #615
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,600
Karma: 2092290
Join Date: Oct 2010
Location: Australia
Device: Kindle Oasis
v1.7.3 Released

Changes in this release:
  • Fix drag/drop into the Goodreads dialog due to change to use of HTTPS url

Thanks to those of you who reported it and KrazyKiwi for correctly guessing the cause...
kiwidude is offline   Reply With Quote
Reply

Tags
calibre, goodreads

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
[GUI Plugin] Search the Internet kiwidude Plugins 422 Yesterday 04:53 PM
[GUI Plugin] Quick Preferences kiwidude Plugins 62 03-16-2024 11:47 PM
[GUI Plugin] Open With kiwidude Plugins 402 03-16-2024 11:44 PM
[GUI Plugin] Clipboard Search kiwidude Plugins 24 03-16-2024 11:06 PM
[GUI Plugin] Book Sync **Deprecated** kiwidude Plugins 111 06-07-2011 07:47 PM


All times are GMT -4. The time now is 05:03 AM.


MobileRead.com is a privately owned, operated and funded community.