Unable to Download Metadata

[Adam KH]

I am running Calibre Portable Version 5.42.0_2.

I have 4 different Calibre Portable installs on a USB Drive. On 21 May I was able to add new books and the Download Metadata worked fine. On 22 May I added new books to 3 of the Portable installs and now Download Metadata no longer works on those three instances.

I get the following error when I get a nil result from the metadata search:

I tried following the instructions deleting the metadata-sources-cache.json from the Calibre configuration directory but this made no difference.

I switched off Dropbox syncing and disabled Kaspersky - again no difference. I installed a VPN and tried accessing from Singapore and the UK but again no difference.

I restored a backup image of one of the portable databases that I keep on a NAS drive. Again this made no difference.

These are large databases so they take a considerable time to restore. Additional I found that some of the Covers were no longer in the faulty Calibre instances.

I tried a fresh install of a new Calibre Portable instance but this still does not return anything on a Metadata search.

At a loss....

Help please.

The error message that occurs is:

calibre, version 5.42.0
ERROR: No matches found: <p>Failed to find any books that match your search. Try making the search <b>less specific</b>. For example, use only the author's last name and a single distinctive word from the title.<p>To see the full log, click "Show details".

Running identify query with parameters:
{'title': 'Hero of the Empire', 'authors': ['Candice Millard'], 'identifiers': {}, 'timeout': 30}
Using plugins: Google (1, 0, 3), Amazon.com (1, 2, 27), Edelweiss (2, 0, 1)
The log from individual plugins is below

****************************** Google (1, 0, 3) ******************************
Found 0 results
Downloading from Google took 0.5406780242919922
Making query: https://books.google.com/books/feeds...ewability=none
Failed to make identify query: 'https://books.google.com/books/feeds/volumes?q=intitle%3AHero+intitle%3Aof+intitle%3AEm pire%2Binauthor%3ACandice+inauthor%3AMillard&max-results=20&start-index=1&min-viewability=none'
Traceback (most recent call last):
File "mechanize\_urllib2_fork.py", line 1236, in do_open
File "http\client.py", line 1255, in request
File "http\client.py", line 1301, in _send_request
File "http\client.py", line 1250, in endheaders
File "http\client.py", line 1010, in _send_output
File "http\client.py", line 950, in send
File "http\client.py", line 1424, in connect
File "ssl.py", line 500, in wrap_socket
File "ssl.py", line 1040, in _create
File "ssl.py", line 1309, in do_handshake
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "calibre\ebooks\metadata\sources\google.py", line 383, in make_query
File "mechanize\_mechanize.py", line 241, in open_novisit
File "mechanize\_mechanize.py", line 287, in _mech_open
File "mechanize\_opener.py", line 193, in open
File "mechanize\_urllib2_fork.py", line 425, in _open
File "mechanize\_urllib2_fork.py", line 414, in _call_chain
File "calibre\utils\browser.py", line 28, in https_open
File "mechanize\_urllib2_fork.py", line 1240, in do_open
urllib.error.URLError: <urlopen error [WinError 10054] An existing connection was forcibly closed by the remote host>

************************************************** ******************************

****************************** Amazon.com (1, 2, 27) ******************************
Found 0 results
Downloading from Amazon.com took 0.5376870632171631
User-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:86.0) Gecko/20100101 Firefox/86.0
Server: auto
Making google query: https://www.google.com/search?q=Hero...www.amazon.com
Plugin Amazon.com failed
Traceback (most recent call last):
File "mechanize\_urllib2_fork.py", line 1236, in do_open
File "http\client.py", line 1255, in request
File "http\client.py", line 1301, in _send_request
File "http\client.py", line 1250, in endheaders
File "http\client.py", line 1010, in _send_output
File "http\client.py", line 950, in send
File "http\client.py", line 1424, in connect
File "ssl.py", line 500, in wrap_socket
File "ssl.py", line 1040, in _create
File "ssl.py", line 1309, in do_handshake
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "calibre\ebooks\metadata\sources\identify.py", line 47, in run
File "calibre\ebooks\metadata\sources\amazon.py", line 1513, in identify
File "calibre\ebooks\metadata\sources\amazon.py", line 1446, in search_search_engine
File "calibre\ebooks\metadata\sources\search_engines.py ", line 288, in google_search
File "calibre\ebooks\metadata\sources\search_engines.py ", line 68, in query
File "mechanize\_mechanize.py", line 241, in open_novisit
File "mechanize\_mechanize.py", line 287, in _mech_open
File "mechanize\_opener.py", line 193, in open
File "mechanize\_urllib2_fork.py", line 425, in _open
File "mechanize\_urllib2_fork.py", line 414, in _call_chain
File "calibre\utils\browser.py", line 28, in https_open
File "mechanize\_urllib2_fork.py", line 1240, in do_open
urllib.error.URLError: <urlopen error [WinError 10054] An existing connection was forcibly closed by the remote host>

************************************************** ******************************

****************************** Edelweiss (2, 0, 1) ******************************
Found 0 results
Downloading from Edelweiss took 0.3200509548187256
Currently Edelweiss returns random books for search queries

************************************************** ******************************
The identify phase took 0.65 seconds
The longest time (0.540678) was taken by: Google
Merging results from different sources
We have 0 merged results, merging took: 0.00 seconds


The error Message that I get is:

[theducks]

The last lines.

Forcibly closed. (they slammed the door in your face)

It could be something as simple as the service was down for maintenance (or someone goofed).

You might have done something to get banned like too many rapid requests.
(Calibre has a (default tweak) setting to normally prevent this. Too many uses can get apps blocked (anti-bot) for being bad netziens. Please don't force rapid fetches from a public service.

[AdamPP]

Originally Posted by theducks

The last lines.

Forcibly closed. (they slammed the door in your face)

It could be something as simple as the service was down for maintenance (or someone goofed).

You might have done something to get banned like too many rapid requests.....

Thanks,
Using a different ID to reply.

This is a problem which occurs sometimes in Cambodia. That is why I tried a VPN. I will wait a day and then try again.

Thanks

[DNSB]

Just as an FYI, using multiple user IDs is not permitted on MobileRead.

[AdamPP]

Originally Posted by DNSB

Just as an FYI, using multiple user IDs is not permitted on MobileRead.

I understand but I do not have the password for the previous User ID and the password reset did not work......

[AdamPP]

I waited 20 hours and tried again but still got the same result.

I have had this in the past with a smaller Library. I got around that by cloning another small library that worked and transferring the books to the cloned library.

I don't know if Calibre Portable has a built in value for the user making the query to Google and Amazon. If not and it uses the IP address then I have to wait until my ISP issues me with a new IP address - I have no control over that.

If it uses my MAC address then I can work around that using a different network connection.

[theducks]

YOUR IP x.x.x.### may be the one blocked
BUT
your ISP (x.x.x) could be on the blocklist or your country could be blocked (firewalled)

ping the target IP
Or it could simply be the sources you chose are NOT licensed for where you are (georestricted)
Some metadata sources can be configured to use other sites.
Select: use the configure source button. Examine each sources setting choices as the differ (I show Amazon which supports their various sites)

[AdamPP]

Originally Posted by theducks

YOUR IP x.x.x.### may be the one blocked
BUT
your ISP (x.x.x) could be on the blocklist or your country could be blocked (firewalled).....

Thanks...

I immediately thought that.

I then tried a VPN which showed my location as Leeds in the UK. This produced no difference and I got the same result.

I then did a disk clean up on all disks "C", "D" and the portable drive.
Having done that I then checked the Privacy settings in Windows 10 and switched off location for the computer and changed the Default location.
I got the same result from google and amazon. Both kicked me off.
I changed the Amazon server to the UK and selected Amazon servers.
I got the same result!

I do not have access to another Windows 10 machine so cannot check on a different machine.

Now at a total loss. I left attempting to download metadata for 24 hours.

I will now try using a different MAC address.

[AdamPP]

I connected through my WiFi which will give a different MAC address.
Again I got bounced....

[AdamPP]

My reply got lost????

1. I disabled Amazon and Google Metadata plug-in
2. Installed Goodreads plug-in
3. Deleted ISBN etc for a book I was able to locate on the Goodreads website.
4. Attempted a Metadata Download.
5. Failure with:
Running identify query with parameters:
{'title': 'November 9', 'authors': ['Colleen Hoover'], 'identifiers': {}, 'timeout': 30}
Using plugins: Goodreads (1, 5, 3), Edelweiss (2, 0, 1)
The log from individual plugins is below

****************************** Goodreads (1, 5, 3) ******************************
Found 0 results
Downloading from Goodreads took 1.2729945182800293
identify - start. title=November 9, authors=['Colleen Hoover'], identifiers={}
Querying: https://www.goodreads.com/search?sea...e=books&search[query]=November+9+Colleen+Hoover
Failed to make identify query: 'https://www.goodreads.com/search?search_type=books&search[query]=November+9+Colleen+Hoover'
Traceback (most recent call last):
File "mechanize\_urllib2_fork.py", line 1236, in do_open
File "http\client.py", line 1255, in request
File "http\client.py", line 1301, in _send_request
File "http\client.py", line 1250, in endheaders
File "http\client.py", line 1010, in _send_output
File "http\client.py", line 950, in send
File "http\client.py", line 1424, in connect
File "ssl.py", line 500, in wrap_socket
File "ssl.py", line 1040, in _create
File "ssl.py", line 1309, in do_handshake
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host

Something is blocking me.

Interestingly, when I enabled Amazon and Google Metadata plugins from preferences It did not attempt to search using Google.

I will now conduct a Kaspersky virus check of the drive.
If that produces nothing I will consider going back to an earlier Acronis C Drive image.

[davidfor]

Originally Posted by AdamPP

My reply got lost????

1. I disabled Amazon and Google Metadata plug-in
2. Installed Goodreads plug-in
3. Deleted ISBN etc for a book I was able to locate on the Goodreads website.
4. Attempted a Metadata Download.
5. Failure with:
Running identify query with parameters:
{'title': 'November 9', 'authors': ['Colleen Hoover'], 'identifiers': {}, 'timeout': 30}
Using plugins: Goodreads (1, 5, 3), Edelweiss (2, 0, 1)
The log from individual plugins is below

****************************** Goodreads (1, 5, 3) ******************************
Found 0 results
Downloading from Goodreads took 1.2729945182800293
identify - start. title=November 9, authors=['Colleen Hoover'], identifiers={}
Querying: https://www.goodreads.com/search?sea...e=books&search[query]=November+9+Colleen+Hoover
Failed to make identify query: 'https://www.goodreads.com/search?search_type=books&search[query]=November+9+Colleen+Hoover'
Traceback (most recent call last):
File "mechanize\_urllib2_fork.py", line 1236, in do_open
File "http\client.py", line 1255, in request
File "http\client.py", line 1301, in _send_request
File "http\client.py", line 1250, in endheaders
File "http\client.py", line 1010, in _send_output
File "http\client.py", line 950, in send
File "http\client.py", line 1424, in connect
File "ssl.py", line 500, in wrap_socket
File "ssl.py", line 1040, in _create
File "ssl.py", line 1309, in do_handshake
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host

Something is blocking me.

Interestingly, when I enabled Amazon and Google Metadata plugins from preferences It did not attempt to search using Google.

I will now conduct a Kaspersky virus check of the drive.
If that produces nothing I will consider going back to an earlier Acronis C Drive image.

The error is a network issue. It is happening outside of calibre. Do you use a proxy? Or does your antivirus do anything for network connections? Or need to give applications permission to connect? If you use the search URL that is in the log in a browser, does it work?

[AdamPP]

Originally Posted by davidfor

The error is a network issue. It is happening outside of calibre. Do you use a proxy? Or does your antivirus do anything for network connections? Or need to give applications permission to connect? If you use the search URL that is in the log in a browser, does it work?

Agree it is a network issue.
1. I don't use a proxy for anything
2. There is no restriction on application permissions. I have the only computer in the house. Everything else is either Apple TV, iPads, iPhones or Android Phoines. I do have multiple NAS drives (QNAP & Synology but all powered off at time of testing)
3. It could be an issue with either my ISP or Modem/Router.
4. I use Kaspersky as my anti-virus but I tested with it disabled.

I will reset my modem router tomorrow morning.

I cannot test on another Windows 10 computer or a different ISP until the weekend.

I could try using my phone as a hotspot using cellular data but my cellular connection where I live is rubbish - I have to go onto the roof of the house for a mediocre connection.

I am not very good at Windows networking but potentially I need to check the networking in the NAS drives (potentially one may be attempting to provide a DNS look-up).

If it is an ISP problem then it means that Calibre will not be usable for me. Where I live in Phnom Penh in Cambodia I only have one ISP available.

If I work step by step... just take my time.

Your guidance is really appreciated.
Thanks

[theducks]

You should only ever have ONE DHCP server on a network. A number of devices you list CAN Offer that service in its config. Just say NO
My Snology CAN be the DHCP, My Access Point CAN be the DHCP. But I leave that job to my Router.
For 99% of home user, the ROUTER is the DHCP server.

OTOH , the DNS service to use, could be set at each device (manual). (not usually if DHCP is used). DNS problems usually result in the 'Taking to long to Respond'/'Address Not Found' on web pages or Pings failing.

[AdamPP]

Originally Posted by theducks

You should only ever have ONE DHCP server on a network. A number of devices you list CAN Offer that service in its config. Just say NO
My Snology CAN be the DHCP, My Access Point CAN be the DHCP. But I leave that job to my Router.
For 99% of home user, the ROUTER is the DHCP server.

OTOH , the DNS service to use, could be set at each device (manual). (not usually if DHCP is used). DNS problems usually result in the 'Taking to long to Respond'/'Address Not Found' on web pages or Pings failing.

I checked all NAS devices (QNAP x 3 & Synology). None have DHCP enabled. My router is a Draytek Vigor 2925 without Wifi (updated to the latest available firmware). This is connected to the ISP's Modem-Router using that Modem-Router as a Gateway.

The ISP's Modem-Router is a Raisecom Epon Onu iscom ht803. It does have a DHCP functionality but i assume cannot be disabled. This modem has been in place since January 2021 and has not caused any issues in the past. The lease time on the current modem is 1 day with dynamic IP address but this seldom changes.

The Draytek is set for supply of the DHCP. Looking at ipconfig on the Windows10 PC it shows that the DHCP Server as 192.168.100.1 which is the Draytek router.

I need to get access to the ISP's modem to check its configuration but that may take a time as their help desk are very bad and you have to persuade them to do a site visit. Plus they only speak Khmer.

The only other issue is the DNS servers. The first is 115.178.24.3 ad the second is Google 8.8.8.8. The first is owned by SingMeng which has now been taken over by Ezecom. Due to Government action reducing the number of ISPs Ezecom have taken over 3 ISPs which the Government forced to close. The Government want to establish a single internet exit point for Cambodia.

Ezecom are struggling with managing the the increased number of users and have multiple outages every day. Although I don't think the DNS servers are the issue I need to consider using alternate Public DNS servers.

Kaspersky virus check on the USB Drive is still running after 15 hours and I expect it to run for a bit longer.

I will change the DNS servers to public servers.

The only change that have occurred around the time of the problem are:
1. I fixed issues with SMB access to the NAS drives allowing SMB V1
2. I installed Facebook messenger app for a virtual meeting (now uninstalled)
3. I tidied up Credentials manager. There are no web credentials.
4. I transferred one old QNAP NAS to an academic institution and spent a lot time ensuring that it was clean. Only data on the NAS was the books and documentaries.

I will reboot the ISP modem when Kasspersky finishes.

[AdamPP]

The issue is with Kaspersky......

I download Version 4.2.17 Portable 32 bit.
Paused Kaspersky protection.
Added a know book.
Downloaded the Metadata for that book successfully.
I deleted the book.

I then enabled Kaspersky
Ran the portable version
Added the book and then attempted the Metadata download.
The Download failed.
I deleted the book.

I deleted that version of portable and reinstalled the same version.
Disabled Kaspersky
Added the book
Attempted a Metadata download
The download failed.

I tried the same with version 5.0.0 and got the same result.

I then tried with Version 5.10.1 and could not get a metadata download with Kaspersky disabled. I did not attempt to carry out the test with Kaspersky enabled.

There seems to be an issue with Kaspersky and a secondary issue of persistence based on major release version number outside of Calibre.