Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre

Notices

Reply
 
Thread Tools Search this Thread
Old 02-26-2017, 06:12 PM   #46
Oddball
Junior Member
Oddball began at the beginning.
 
Posts: 7
Karma: 10
Join Date: Nov 2016
Device: Kindle Desktop
I hate to say this, but maybe its time to figure out a way to create a site that images amazon without the captcha for the exclusive use of bots to use. MAYBE it is possible to use Wayback machine for that. A friend of mine data mined a site and create a mirror image of the original site. would just require a large storage of at least a TB or more to mirror. Just a thought.

Last edited by Oddball; 02-26-2017 at 06:16 PM.
Oddball is offline   Reply With Quote
Advert
Old 02-26-2017, 06:32 PM   #47
PeterT
Grand Sorcerer
PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.
 
PeterT's Avatar
 
Posts: 9,634
Karma: 56440393
Join Date: Nov 2007
Location: Toronto
Device: Wife: Touch, Arc, Vox Me: Nexus 7, Glo
I seriously doubt Amazon would allow that
PeterT is offline   Reply With Quote
Old 02-26-2017, 07:32 PM   #48
Oddball
Junior Member
Oddball began at the beginning.
 
Posts: 7
Karma: 10
Join Date: Nov 2016
Device: Kindle Desktop
Quote:
Originally Posted by PeterT View Post
I seriously doubt Amazon would allow that
well truthfully they are already and have been apart of Archive.org since 2000 at least.....it is the one way i find info on some eBook they have discontinued.

http://amazon.com

Saved 52,539 times between December 12, 1998 and February 26, 2017.
Oddball is offline   Reply With Quote
Old 02-26-2017, 08:27 PM   #49
PeterT
Grand Sorcerer
PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.
 
PeterT's Avatar
 
Posts: 9,634
Karma: 56440393
Join Date: Nov 2007
Location: Toronto
Device: Wife: Touch, Arc, Vox Me: Nexus 7, Glo
Fine go and make a metadata plugin to access archive.org's copy of Amazon then
PeterT is offline   Reply With Quote
Old 02-26-2017, 10:10 PM   #50
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 30,179
Karma: 7723335
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
It's not quite as simple as using a mirror. The problem is metadata plugins require a way to search for title/author/isbn/etc. So in addition to a mirror you'd need to implement search as well.

So directly using something like archive.org is not possible. What one can do is probably use google's sitesearch for archive.org. Though IIRC gogole does not like bots using their search either, so you'd likely run into the same problems as with amazon, eventually.
kovidgoyal is offline   Reply With Quote
Advert
Old 02-28-2017, 12:19 PM   #51
jhowell
Wizard
jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.
 
jhowell's Avatar
 
Posts: 1,249
Karma: 11034281
Join Date: Nov 2011
Location: Florida
Device: iPad Air 2, Kindle PW2, Nexus 7, Fire 5th Gen
In my personal experience it seems that Amazon doesn't bother users again for quite a while once they answer a captcha, as long as cookies are maintained.

Perhaps a solution would be to present the captcha to the user for solution, proving the non-botness of the requester along with preserving the Amazon cookies in a more permanent fashion.

ETA: The function of making a request to Amazon and having the user solve the captcha could be a UI plugin, separate from the Amazon metadata plugin, as long as they share cookies.

Last edited by jhowell; 02-28-2017 at 12:25 PM.
jhowell is offline   Reply With Quote
Old 02-28-2017, 12:44 PM   #52
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 30,179
Karma: 7723335
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
That's awfully fiddly, and probably if amazon detects that bots resume operations after answering a captcha on the same IP it will escalate to banning the IP with HTTP 503 or the like. After all, it is actually fairly trivial to solve captchas programmatically, the problem with doing that is that the next step is getting the IP banned, at least that is how I recall most bot detection algorithms work. A bit of googling suggests that is indeed how they work.
kovidgoyal is offline   Reply With Quote
Old 02-28-2017, 01:32 PM   #53
jhowell
Wizard
jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.
 
jhowell's Avatar
 
Posts: 1,249
Karma: 11034281
Join Date: Nov 2011
Location: Florida
Device: iPad Air 2, Kindle PW2, Nexus 7, Fire 5th Gen
I definitely do not want my IP address banned by Amazon. It is too bad that they are taking these steps to block programmatic access to their site.
jhowell is offline   Reply With Quote
Old 02-28-2017, 10:43 PM   #54
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 30,179
Karma: 7723335
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Looking over the code for the amazon plugin, I did find one place where the plugin was making a handful of simultaneous requests to amazon, maybe that was trigerring the bot detection. I've inserted some sleeps -- this will make downloading slower, but hopefully more reliable.
kovidgoyal is offline   Reply With Quote
Old 02-28-2017, 11:09 PM   #55
Oddball
Junior Member
Oddball began at the beginning.
 
Posts: 7
Karma: 10
Join Date: Nov 2016
Device: Kindle Desktop
I just want to thank you for all of your massive work on this wonderful program.

So, the current plugin hits the site multiple times, much like the way windows hammers network shares?
Oddball is offline   Reply With Quote
Old 02-28-2017, 11:25 PM   #56
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 30,179
Karma: 7723335
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
No not really, the plugin actually hits the site less times than a typical browser request (browsers typically make between 5-10 simulataneous requests for a single domain) I've just further reduced to no more than one request a second.
kovidgoyal is offline   Reply With Quote
Old 03-01-2017, 09:05 AM   #57
Oddball
Junior Member
Oddball began at the beginning.
 
Posts: 7
Karma: 10
Join Date: Nov 2016
Device: Kindle Desktop
This might sound like a moronic question but, what is the url to click on the CAPTCHA? i have looked but cant find it and amazons error is just as helpful as ANY from Microsoft.
Oddball is offline   Reply With Quote
Old 03-01-2017, 09:15 AM   #58
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 30,179
Karma: 7723335
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
There is no URL basically accessing any page will redirect to the captcha. But it may or may nt happen with your regular browser since that will have a different set of cookies/profile.
kovidgoyal is offline   Reply With Quote
Old 03-02-2017, 09:57 AM   #59
audeojude
Enthusiast
audeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshesaudeojude can read faster than his screen refreshes
 
Posts: 42
Karma: 14170
Join Date: Oct 2011
Device: kindle 1
So I've mentioned on another thread having this problem on and off for the last few months. In the last few weeks amazon has become unusable for me to download metadata. I have changed ip's (even what country my ip is in), what amazon site it gets metadata from, all kinds of things I have tried. once in a while it will give me 3 or 4 books but most often just days of doesn't work. I had gotten to copying from browser to comments field and typing in tags and series information by hand

On reading this thread I saw the mention of the goodreads plugin and got it and started using it. Since it is owned by amazon I'm thinking all the KU books and self published will show up there also. I just did a search for 35 books and got back 27 results. So it is working better than amazon for me at the moment.


I also want to add my thanks for calibre.. I have kicked in money a few times and will again. I use it a lot. I suggest for all of you that really do use it to kick back on paypal or other way to kovid a few dollars. I have over the years sent 10 to 25 dollars each time. I have some comercial software I have to use and have payed hundreds of dollars for. It doesn't work as well as this, doesn't get updated as often and bugs fixed as quickly. It really is worth our time to let kovid know how much we appreciate it. I think I will go do that right now Words are nice but actions speak greatly. I maintain a community website and when members send me a few dollars in thanks for the time and effort I put in it really means a lot to me.

Last edited by audeojude; 03-02-2017 at 10:03 AM.
audeojude is offline   Reply With Quote
Old 03-02-2017, 10:03 AM   #60
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 30,179
Karma: 7723335
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
I'm actualy currently working on modifying the amazon plugin to get the results from various search engine web caches, bing, google, wayback machine as well as amazon itself. Probably make it configurable so if one source stops working you can switch to another. That combined with the changes I made to further reduce query frequency should hopefully get it working again. If it doesn't then I'm afraid I am truly out of ideas.
kovidgoyal is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Aussies launch anti-CAPTCHA petition. Is it time to kill CAPTCHAs? Alexander Turcic Lounge 30 08-16-2013 07:32 PM
Bug in Kobo processing of epub files causing hang in "Processing content" BensonBear Kobo Reader 21 12-21-2012 05:47 AM
Get books bug for Amazon UK rustleg Library Management 1 10-21-2012 01:25 PM


All times are GMT -4. The time now is 11:50 AM.


MobileRead.com is a privately owned, operated and funded community.