Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre

Notices

Reply
 
Thread Tools Search this Thread
Old 03-02-2017, 10:31 AM   #61
audeojude
Enthusiast
audeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshes
 
Posts: 42
Karma: 14170
Join Date: Oct 2011
Device: kindle 1
That would be cool but I have the feeling that it is going to get harder and harder.

Amazon is my goto place for books because that is where all the indie authors are as well as ease of purchase, I follow so many authors that it would be impossible for me to do so easily without somewhere like amazon to track new releases and purchase there. I have to admit that as a business I am not terribly happy with some of their business practices, lobbying and other such things.

as far as this issue goes, well.. if they have enough people hitting them for bulk metadata searches that is is affecting their performance from time to time then I can see their point. I don't like it because I am a customer and am just doing it mostly for books purchased there. However I have noticed times in the last year when their website has slowed down significantly between when I purchase a book and it shows up as available (Average is usually 2 or 3 seconds. I have seen it take minutes sometimes though) . To me this speaks of a big hit on their database's. I imagine they are starting to get ruthless about anything outside their core money making practices that affect performance to get cut or stopped.
audeojude is offline   Reply With Quote
Advert
Old 03-02-2017, 10:51 AM   #62
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 31,295
Karma: 8154029
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
That's the nice thing about using the search engine caches, it means amazon is not touched at all (well covers are still downloaded from amazons servers, but that's it. Expensive queries that touch databases are avoided). This does mean that ratings for some books maybe a little out of date.

calibre's metadata fetching system is not really a bot. It runs only when asked to run by an actual human and makes targeted queries for the book in question. It does not indiscriminately scrape sites. It also goes out of its way to make sure it throttles all interaction with the servers it queries so as not to overload them.

The other default metadata plugin, that queries google books has never had any issues with google, it's a shame that Amazon does not distinguish between responsible uses of their website and bad actors. But, such is the world, we do what we can...
kovidgoyal is offline   Reply With Quote
Old 03-02-2017, 11:25 AM   #63
audeojude
Enthusiast
audeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshes
 
Posts: 42
Karma: 14170
Join Date: Oct 2011
Device: kindle 1
I wonder if you directly approached amazon and asked if you could use your own browser agent id that would allow calibre to work or maybe if for those with amazon accounts some way for the metadata plugin to login with that users account to not have the captcha get in the way. I'm not sure how they would view calibre though as I think they would rather you left your books on their cloud other than when it is on your kindle. Not to mention the DRM issues with the third party plugins available for calibre.

On second thought they would probably love to have you approach them and sign a contract restricting those features from being available to make metadata work. I would be scared as shit to ever sign a contract of any kind with them. I have the feeling they would eat you alive on the backend. They seem to be that kind of company.
audeojude is offline   Reply With Quote
Old 03-02-2017, 12:26 PM   #64
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 31,295
Karma: 8154029
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
You cant do that. Even if they were amenable, there's nothing to stop malicious actors from using the same uuser agent. And since calibre is open source, there is no way to have some kind of secret known only to calibre and amazon.
kovidgoyal is offline   Reply With Quote
Old 03-02-2017, 04:42 PM   #65
audeojude
Enthusiast
audeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshes
 
Posts: 42
Karma: 14170
Join Date: Oct 2011
Device: kindle 1
Yeah I had that thought after posting the idea. Sometimes you just can't win!
audeojude is offline   Reply With Quote
Old 03-02-2017, 05:37 PM   #66
peekpoke
Member
peekpoke began at the beginning.
 
Posts: 22
Karma: 10
Join Date: Dec 2013
Device: Kindle/Android
What about using the official Amazon API, like this?
http://docs.aws.amazon.com/AWSEComme...kupbyISBN.html

Or this free example for searching?
https://aws.amazon.com/customerapps/666
http://www.theeasystore.com/asin_lookup_tool.php

Perhaps an option to read cookies from a browser, or put in Amazon or AWS login information?
peekpoke is offline   Reply With Quote
Old 03-02-2017, 06:24 PM   #67
JSWolf
Resident Curmudgeon
JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.
 
JSWolf's Avatar
 
Posts: 48,562
Karma: 39003673
Join Date: Nov 2006
Location: Roslindale, Massachusetts
Device: Kobo Aura H2O, Sony PRS-650, Sony PRS-T1, nook STR, iPad 4, iPhone 5
What about Overdrive?
JSWolf is offline   Reply With Quote
Old 03-02-2017, 07:17 PM   #68
davidfor
Grand Sorcerer
davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.
 
Posts: 12,636
Karma: 19973288
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo: Touch, Glo, Aura H2O, Glo HD, Aura ONE
@Kovid: Is it possible the problem is that the metadata download is not accessing other resources on the pages involved? Not doing that could easily be seen as a sign that of a bot. Of course, working out which resource is the important one is the problem.
davidfor is offline   Reply With Quote
Old 03-02-2017, 09:44 PM   #69
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 31,295
Karma: 8154029
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
@peekpoke The official amazon apis require an API key, every calibre user would need to get their own api key. That is too painful, at least for me to maintain. The default metadata plugins need to work out of the box, with no need for configuration.

But certainly, if it turns out that I cannot make the plugin work, then it's fine for a new plugin to be written using the API, rather like the existing isbndb one.

@davidfor Possibly, but I doubt it. If that were the case defeating it would be fairly esay for determined scrapers and moreover, defeating it would actually greatly increase load on the servers. Bit of a footgun strategy
kovidgoyal is offline   Reply With Quote
Old 03-02-2017, 09:46 PM   #70
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 31,295
Karma: 8154029
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
@JSWolf certainly an option to explore if I fail at fixing this plugin.
kovidgoyal is offline   Reply With Quote
Old 03-10-2017, 10:37 PM   #71
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 31,295
Karma: 8154029
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
version 2.81 can now download amazon metadata from various sources (configured via Preferences->Metadata sources) so that should help with this issue
kovidgoyal is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Aussies launch anti-CAPTCHA petition. Is it time to kill CAPTCHAs? Alexander Turcic Lounge 30 08-16-2013 07:32 PM
Bug in Kobo processing of epub files causing hang in "Processing content" BensonBear Kobo Reader 21 12-21-2012 05:47 AM
Get books bug for Amazon UK rustleg Library Management 1 10-21-2012 01:25 PM


All times are GMT -4. The time now is 07:43 AM.


MobileRead.com is a privately owned, operated and funded community.