03-02-2017, 10:31 AM | #61 |
Connoisseur
Posts: 66
Karma: 14170
Join Date: Oct 2011
Device: kindle 1
|
That would be cool but I have the feeling that it is going to get harder and harder.
Amazon is my goto place for books because that is where all the indie authors are as well as ease of purchase, I follow so many authors that it would be impossible for me to do so easily without somewhere like amazon to track new releases and purchase there. I have to admit that as a business I am not terribly happy with some of their business practices, lobbying and other such things. as far as this issue goes, well.. if they have enough people hitting them for bulk metadata searches that is is affecting their performance from time to time then I can see their point. I don't like it because I am a customer and am just doing it mostly for books purchased there. However I have noticed times in the last year when their website has slowed down significantly between when I purchase a book and it shows up as available (Average is usually 2 or 3 seconds. I have seen it take minutes sometimes though) . To me this speaks of a big hit on their database's. I imagine they are starting to get ruthless about anything outside their core money making practices that affect performance to get cut or stopped. |
03-02-2017, 10:51 AM | #62 |
creator of calibre
Posts: 43,739
Karma: 22446736
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
That's the nice thing about using the search engine caches, it means amazon is not touched at all (well covers are still downloaded from amazons servers, but that's it. Expensive queries that touch databases are avoided). This does mean that ratings for some books maybe a little out of date.
calibre's metadata fetching system is not really a bot. It runs only when asked to run by an actual human and makes targeted queries for the book in question. It does not indiscriminately scrape sites. It also goes out of its way to make sure it throttles all interaction with the servers it queries so as not to overload them. The other default metadata plugin, that queries google books has never had any issues with google, it's a shame that Amazon does not distinguish between responsible uses of their website and bad actors. But, such is the world, we do what we can... |
Advert | |
|
03-02-2017, 11:25 AM | #63 |
Connoisseur
Posts: 66
Karma: 14170
Join Date: Oct 2011
Device: kindle 1
|
I wonder if you directly approached amazon and asked if you could use your own browser agent id that would allow calibre to work or maybe if for those with amazon accounts some way for the metadata plugin to login with that users account to not have the captcha get in the way. I'm not sure how they would view calibre though as I think they would rather you left your books on their cloud other than when it is on your kindle. Not to mention the DRM issues with the third party plugins available for calibre.
On second thought they would probably love to have you approach them and sign a contract restricting those features from being available to make metadata work. I would be scared as shit to ever sign a contract of any kind with them. I have the feeling they would eat you alive on the backend. They seem to be that kind of company. |
03-02-2017, 12:26 PM | #64 |
creator of calibre
Posts: 43,739
Karma: 22446736
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
You cant do that. Even if they were amenable, there's nothing to stop malicious actors from using the same uuser agent. And since calibre is open source, there is no way to have some kind of secret known only to calibre and amazon.
|
03-02-2017, 04:42 PM | #65 |
Connoisseur
Posts: 66
Karma: 14170
Join Date: Oct 2011
Device: kindle 1
|
Yeah I had that thought after posting the idea. Sometimes you just can't win!
|
Advert | |
|
03-02-2017, 05:37 PM | #66 |
Enthusiast
Posts: 37
Karma: 10
Join Date: Dec 2013
Device: Kindle/Android
|
What about using the official Amazon API, like this?
http://docs.aws.amazon.com/AWSEComme...kupbyISBN.html Or this free example for searching? https://aws.amazon.com/customerapps/666 http://www.theeasystore.com/asin_lookup_tool.php Perhaps an option to read cookies from a browser, or put in Amazon or AWS login information? |
03-02-2017, 06:24 PM | #67 |
Resident Curmudgeon
Posts: 73,510
Karma: 126422064
Join Date: Nov 2006
Location: Roslindale, Massachusetts
Device: Kobo Libra 2, Kobo Aura H2O, PRS-650, PRS-T1, nook STR, PW3
|
What about Overdrive?
|
03-02-2017, 07:17 PM | #68 |
Grand Sorcerer
Posts: 24,908
Karma: 47303748
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
|
@Kovid: Is it possible the problem is that the metadata download is not accessing other resources on the pages involved? Not doing that could easily be seen as a sign that of a bot. Of course, working out which resource is the important one is the problem.
|
03-02-2017, 09:44 PM | #69 |
creator of calibre
Posts: 43,739
Karma: 22446736
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
@peekpoke The official amazon apis require an API key, every calibre user would need to get their own api key. That is too painful, at least for me to maintain. The default metadata plugins need to work out of the box, with no need for configuration.
But certainly, if it turns out that I cannot make the plugin work, then it's fine for a new plugin to be written using the API, rather like the existing isbndb one. @davidfor Possibly, but I doubt it. If that were the case defeating it would be fairly esay for determined scrapers and moreover, defeating it would actually greatly increase load on the servers. Bit of a footgun strategy |
03-02-2017, 09:46 PM | #70 |
creator of calibre
Posts: 43,739
Karma: 22446736
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
@JSWolf certainly an option to explore if I fail at fixing this plugin.
|
03-10-2017, 10:37 PM | #71 |
creator of calibre
Posts: 43,739
Karma: 22446736
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
version 2.81 can now download amazon metadata from various sources (configured via Preferences->Metadata sources) so that should help with this issue
|
Thread Tools | Search this Thread |
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
Aussies launch anti-CAPTCHA petition. Is it time to kill CAPTCHAs? | Alexander Turcic | Lounge | 30 | 08-16-2013 07:32 PM |
Bug in Kobo processing of epub files causing hang in "Processing content" | BensonBear | Kobo Reader | 21 | 12-21-2012 05:47 AM |
Get books bug for Amazon UK | rustleg | Library Management | 1 | 10-21-2012 01:25 PM |