Can only download Amazon metadata while connected to VPN

josephyw · 03-15-2021, 03:25 PM

Hi all,

I've recently encountered a strange issue with attempting to download metadata through Calibre in addition to the Kobo metadata plugin suddenly not working (which I've documented in this post in the appropriate plugin thread).

Whenever I attempt to download metadata, I now only get Google Books results. Amazon is unable to retrieve data. I've clipped the Amazon portion of the log below:

Spoiler:

However, I had the random idea of using a VPN and tried to do the search again, and it surprisingly worked. The results of that search, on the same book but instead connected to a Windscribe VPN server, are below:

Spoiler:

Does anybody have any insight into what might be going on—and how I can fix it moving forward? Thank you for any time and feedback you can offer!

Best wishes,
Joseph

kovidgoyal · 03-15-2021, 10:29 PM

presumably your ISP is filtering/modifying the html you get. Which your VPN is bypassing.

josephyw · 03-16-2021, 07:17 AM

Thank you so much for your response, Mr. Goyal. Do you know if there's a way to do more testing or fix this issue, beyond signing into a VPN whenever I want to download metadata?

kovidgoyal · 03-16-2021, 07:50 AM

No not that I can see. calibre can only work with the html it receives, it has no control over what is sent r modified in transit.

josephyw · 03-20-2021, 08:58 AM

Thank you so much for your feedback. Do you happen to know of any tools that would allow me to compare the HTML being received through my ISP to the "pure" HTML, to see how it's being modified?

theducks · 03-20-2021, 12:24 PM

Quote:

Originally Posted by josephyw

Thank you so much for your feedback. Do you happen to know of any tools that would allow me to compare the HTML being received through my ISP to the "pure" HTML, to see how it's being modified?

If your ISP is modifying it, how would you get the unmodified version to compare against ?
If you are not in one of the places that Firewall US content, I would suspect you have Adware/Malware on your system

03-15-2021, 03:25 PM	#1
josephyw Member Posts: 14 Karma: 10 Join Date: Mar 2021 Device: Kobo Libra H2O	Can only download Amazon metadata while connected to VPN Hi all, I've recently encountered a strange issue with attempting to download metadata through Calibre in addition to the Kobo metadata plugin suddenly not working (which I've documented in this post in the appropriate plugin thread). Whenever I attempt to download metadata, I now only get Google Books results. Amazon is unable to retrieve data. I've clipped the Amazon portion of the log below: Spoiler: **************************** Amazon.com (1, 2, 15) ************************** Found 0 results Downloading from Amazon.com took 3.2727837562561035 User-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36 Server: auto Making google query: https://www.google.com/search?q=%289...www.amazon.com Failed to find any results on results page, with title: (9781526121806 or 978-1526121806) site:www.amazon.com - Google Search No search engine results for terms: (9781526121806 OR 978-1526121806) Trying the bing search engine instead Making bing query: https://www.bing.com/search?q=%28978...www.amazon.com Failed to find any results on results page, with title: (9781526121806 or 978-1526121806) site:www.amazon.com - Bing No search engine results for terms: (9781526121806 OR 978-1526121806) No matches found with identifiers, retrying using only title and authors. Query: ['(9781526121806', 'OR', '978-1526121806)'] User-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36 Server: auto Making google query: https://www.google.com/search?q=Begi...www.amazon.com Ignoring div with no descendant Ignoring div with no descendant Ignoring div with no descendant Ignoring div with no descendant Ignoring div with no descendant Ignoring div with no descendant Ignoring div with no descendant Ignoring div with no descendant Ignoring div with no descendant Ignoring div with no descendant Failed to find any results on results page, with title: Beginning Theory Peter Barry site:www.amazon.com - Google Search No search engine results for terms: Beginning Theory Peter Barry Trying the bing search engine instead Making bing query: https://www.bing.com/search?q=Beginn...www.amazon.com Failed to find any results on results page, with title: Beginning Theory Peter Barry site:www.amazon.com - Bing No search engine results for terms: Beginning Theory Peter Barry No matches found with query: ['Beginning', 'Theory', 'Peter', 'Barry'] ********************************************** ************************** However, I had the random idea of using a VPN and tried to do the search again, and it surprisingly worked. The results of that search, on the same book but instead connected to a Windscribe VPN server, are below: Spoiler: ************************** Amazon.com (1, 2, 15) ************************** Found 2 results Downloading from Amazon.com took 7.480014085769653 --- Title : Beginning Theory: An Introduction to Literary and Cultural Theory: Fourth Edition Author(s) : Peter Barry Rating : 2.3 Identifiers : amazon:1526121794 Comments : Beginning theory has been helping students navigate through the thickets of literary and cultural theory for over two decades. This new and expanded fourth edition continues to offer readers the best single-volume introduction to the field. The bewildering variety of approaches, theorists and technical language is lucidly and expertly unravelled. Unlike many books which assume certain positions about the critics and the theories they represent, Beginning theory allows readers to develop their own ideas once first principles and concepts have been grasped. The book has been updated for this edition and includes a new introduction, expanded chapters, and an overview of the subject ('Theory after "Theory"') which maps the arrival of new 'isms' since the second edition appeared in 2002 and the third edition in 2009. --- Title : Beginning Theory: An Introduction to Literary and Cultural Theory Author(s) : Peter Barry Rating : 2.2 Identifiers : amazon:0719079276 Comments : This book has been helping students navigate through the thickest of literary and cultural theory for well over a decade now. This new and expanded third edition continues to offer students and readers the best one-volume introduction to the field. The bewildering variety of approaches, theorists and technical language is lucidly and expertly unravelled. Unlike many books which assume certain positions about the critics and the theories they represent, Peter Barry allows readers to develop their own ideas once first principles and concepts have been grasped. The book has been updated and includes two new chapters, one of which (Literary theory -- a history in ten events) innovatively surveys the course of theory, while the other (Theory after ‘Theory’) maps the arrival of new 'isms' since the second edition appeared in 2002. User-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36 Server: auto Making google query: https://www.google.com/search?q=%289...www.amazon.com Failed to find any results on results page, with title: (9781526121806 or 978-1526121806) site:www.amazon.com - Google Search No search engine results for terms: (9781526121806 OR 978-1526121806) Trying the bing search engine instead Making bing query: https://www.bing.com/search?q=%28978...www.amazon.com Failed to find any results on results page, with title: (9781526121806 or 978-1526121806) site:www.amazon.com - Bing No search engine results for terms: (9781526121806 OR 978-1526121806) No matches found with identifiers, retrying using only title and authors. Query: ['(9781526121806', 'OR', '978-1526121806)'] User-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36 Server: auto Making google query: https://www.google.com/search?q=Begi...www.amazon.com Ignoring div with no descendant Ignoring div with no descendant Ignoring div with no descendant Ignoring div with no descendant Ignoring div with no descendant Ignoring div with no descendant Ignoring div with no descendant Ignoring div with no descendant Ignoring div with no descendant Ignoring div with no descendant Failed to find any results on results page, with title: Beginning Theory Peter Barry site:www.amazon.com - Google Search No search engine results for terms: Beginning Theory Peter Barry Trying the bing search engine instead Making bing query: https://www.bing.com/search?q=Beginn...www.amazon.com Getting details from: https://cc.bingj.com/cache.aspx?q=Be...zZWyH95TzzuOs4 Failed to find product description for url: 'https://cc.bingj.com/cache.aspx?q=Beginning+Theory+Peter+Barry+site%3Aw ww.amazon.com&d=4754773802222791&mkt=en-US&setlang=en-US&w=hnC06KNnMx1yOK-VtwzZWyH95TzzuOs4' Getting details from: https://cc.bingj.com/cache.aspx?q=Be...KZU2jSISCdVyiI Failed to find product description for url: 'https://cc.bingj.com/cache.aspx?q=Beginning+Theory+Peter+Barry+site%3Aw ww.amazon.com&d=4797783598894854&mkt=en-US&setlang=en-US&w=Gm3eKbiwtr2Fx6McZLKZU2jSISCdVyiI' Getting details from: https://cc.bingj.com/cache.aspx?q=Be...HynOhmNJVFbeGV Failed to find product description for url: 'https://cc.bingj.com/cache.aspx?q=Beginning+Theory+Peter+Barry+site%3Aw ww.amazon.com&d=4770776842767927&mkt=en-US&setlang=en-US&w=NZV29war0cV5f5_h2CHynOhmNJVFbeGV' ********************************************** **************************** Does anybody have any insight into what might be going on—and how I can fix it moving forward? Thank you for any time and feedback you can offer! Best wishes, Joseph

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Calibre 3.20: no metadata download from amazon	login_ms	Calibre	13	04-15-2018 01:35 PM
[metadata download]Is it possible to get book price from amazon?	lousi	Calibre	18	07-20-2017 06:36 PM
Download metadata from Amazon broke with last version?	squeebook	Calibre	6	03-18-2017 11:34 PM
Problem: download metadata from german Amazon site	ratanplan	Library Management	12	09-25-2015 06:24 AM

03-15-2021, 10:29 PM	#2
kovidgoyal creator of calibre Posts: 45,318 Karma: 27111242 Join Date: Oct 2006 Location: Mumbai, India Device: Various	presumably your ISP is filtering/modifying the html you get. Which your VPN is bypassing.

03-16-2021, 07:17 AM	#3
josephyw Member Posts: 14 Karma: 10 Join Date: Mar 2021 Device: Kobo Libra H2O	Thank you so much for your response, Mr. Goyal. Do you know if there's a way to do more testing or fix this issue, beyond signing into a VPN whenever I want to download metadata?

03-16-2021, 07:50 AM	#4
kovidgoyal creator of calibre Posts: 45,318 Karma: 27111242 Join Date: Oct 2006 Location: Mumbai, India Device: Various	No not that I can see. calibre can only work with the html it receives, it has no control over what is sent r modified in transit.

03-20-2021, 08:58 AM	#5
josephyw Member Posts: 14 Karma: 10 Join Date: Mar 2021 Device: Kobo Libra H2O	Thank you so much for your feedback. Do you happen to know of any tools that would allow me to compare the HTML being received through my ISP to the "pure" HTML, to see how it's being modified?

Advert

Advert