Metadata fetching extremely slow on Linux

[tailred]

Hi

When I tried fetching metadata today, it was extremely slow (about 90s). Trying to only enable 1 source at a time seemed to show that all sources tested (Amazon, Google, Goodreads) had the same issue, and even trying the flatpak version showed the same issue. I also tried fetching metadata from the same sources for the same book on a Windows 10 VM running on the same machine, and it ran much faster (less than 10 seconds).

My internet is around 200mbits symmetrical so it unlikely to be the issue, and swapping DNS servers didn't seem to make any difference either. I also tried changing some of the metadata download settings, but they didn't seem to make any difference either.

Here is the download log from Calibre Linux:

Running identify query with parameters: 
{'title': 'Irina: The Vampire Cosmonaut Vol. 5', 'authors': ['Keisuke Makino'], 'identifiers': {'isbn': '9798888438329'}, 'timeout': 30} 
Using plugins: Goodreads (1, 7, 9), Google (1, 1, 1), Amazon.com (1, 3, 5) 
The log from individual plugins is below 
Not waiting any longer for more results. Still running sources: 
	Amazon.com 

****************************** Goodreads (1, 7, 9) ****************************** 
Found 1 results 
Downloading from Goodreads took 84.98996186256409 


--- 
Title               : Irina: The Vampire Cosmonaut (Light Novel) Vol. 5
Author(s)           : Keisuke Makino & Karei
Publisher           : Airship
Series              : Irina: The Vampire Cosmonaut Light Novel #5
Languages           : eng
Rating              : 2
Published           : 2019-08-21T07:00:00+00:00
Identifiers         : goodreads:175423772, isbn:9798888438329
Comments            : 
Lev Leps, now vice-director of LAIKA44's training center, is raising the next generation of cosmonauts alongside vampire instructor Irina Luminesk. They soon discover that the engagement between fellow cosmonauts Mikhail and Roza is a farce concocted by the government to smother news of the space development standstill! Enraged at being pawns in a game of national pride, Lev and Irina struggle to move forward. A tragedy looming in the not-so-distant future might just push them over the brink…
 
identify - start. title=Irina: The Vampire Cosmonaut Vol. 5, authors=['Keisuke Makino'], identifiers={'isbn': '9798888438329'}
get_goodreads_id_from_identifiers - isbn=9798888438329
get_goodreads_id_using_api - identifiers=9798888438329
Querying using autocomplete API: https://www.goodreads.com/book/auto_complete?format=json&q=9798888438329
JSON Result: b'[{"imageUrl":"https://i.gr-assets.com/images/S/compressed.photo.goodreads.com/books/1686212853i/175423772._SX50_.jpg","bookId":"175423772","workId":"76761931","bookUrl":"/book/show/175423772-irina","from_search":true,"from_srp":true,"qid":"PolC4Wsrov","rank":1,"title":"Irina: The Vampire Cosmonaut (Light Novel) Vol. 5","bookTitleBare":"Irina: The Vampire Cosmonaut (Light Novel) Vol. 5","numPages":203,"avgRating":"4.00","ratingsCount":1,"author":{"id":20098676,"name":"Keisuke Makino","isGoodreadsAuthor":false,"profileUrl":"https://www.goodreads.com/author/show/20098676.Keisuke_Makino","worksListUrl":"https://www.goodreads.com/author/list/20098676.Keisuke_Makino"},"kcrPreviewUrl":null,"description":{"html":"Lev Leps, now vice-director of LAIKA44\\u0026apos;s training center, is raising the next generation of cosmonauts alongside vampire instructor Irina Luminesk. They soon discover that the\xe2\x80\xa6","truncated":true,"fullContentUrl":"https://www.goodreads.com/book/show/175423772-irina"}}]'
Result using autocomplete API: 175423772
Get details attempt #1
Goodreads book url: 'https://www.goodreads.com/book/show/175423772'
Trying to parse book json for 2022 web page format
Json script node found, page in 2022+ html format
Empty apolloState node, will scrape page instead
No book_json found in this response, retrying for another response
Get details attempt #2
Goodreads book url: 'https://www.goodreads.com/book/show/175423772'
Trying to parse book json for 2022 web page format
Json script node found, page in 2022+ html format
Empty apolloState node, will scrape page instead
No book_json found in this response, retrying for another response
Get details attempt #3
Goodreads book url: 'https://www.goodreads.com/book/show/175423772'
Trying to parse book json for 2022 web page format
Json script node found, page in 2022+ html format
Empty apolloState node, will scrape page instead
No book_json found in this response, retrying for another response
Get details attempt #4
Goodreads book url: 'https://www.goodreads.com/book/show/175423772'
Trying to parse book json for 2022 web page format
Json script node found, page in 2022+ html format
parse_title:  Irina: The Vampire Cosmonaut (Light Novel) Vol. 5
parse_authors - author=Keisuke Makino type=Contributor
parse_authors - author=KAREI type=Contributor
parse_details - goodreads_id: 175423772, mi: Title               : Irina: The Vampire Cosmonaut (Light Novel) Vol. 5
Author(s)           : Keisuke Makino & KAREI
parse_series: series_name='Irina: The Vampire Cosmonaut Light Novel', series_index='5.0'
parse_isbn:  9798888438329
parse_asin:  None
parse_rating:  4.0
parse_comments:  
Lev Leps, now vice-director of LAIKA44's training center, is raising the next generation of cosmonauts alongside vampire instructor Irina Luminesk. They soon discover that the engagement between fellow cosmonauts Mikhail and Roza is a farce concocted by the government to smother news of the space development standstill! Enraged at being pawns in a game of national pride, Lev and Irina struggle to move forward. A tragedy looming in the not-so-distant future might just push them over the brink…

Visiting image url: https://images-na.ssl-images-amazon.com/images/S/compressed.photo.goodreads.com/books/1686212853i/175423772.jpg
parse_tags: 
parse_publisher:  Airship
parse_publish_date: 2019-08-21 07:00:00+00:00 (First published)
parse_language:  eng 

******************************************************************************** 

****************************** Google (1, 1, 1) ****************************** 
Found 1 results 
Downloading from Google took 60.5926399230957 


--- 
Title               : Irina: The Vampire Cosmonaut (Light Novel) Vol. 5
Author(s)           : Keisuke Makino & Karei
Publisher           : Seven Seas Entertainment
Tags                : Comics & Graphic Novels, Manga, Light Novel, Fiction, Science Fiction, Military, Alternative History
Languages           : eng
Published           : 2023-06-08T20:07:45.218421+00:00
Identifiers         : google:Gc6_EAAAQBAJ, isbn:9798888438329
Comments            : 
Lev Leps, now vice-director of LAIKA44's training center, is raising the next generation of cosmonauts alongside vampire instructor Irina Luminesk. They soon discover that the engagement between fellow cosmonauts Mikhail and Roza is a farce concocted by the government to smother news of the space development standstill! Enraged at being pawns in a game of national pride, Lev and Irina struggle to move forward. A tragedy looming in the not-so-distant future might just push them over the brink...
 
Making query: https://books.google.com/books/feeds/volumes?q=isbn%3A9798888438329&max-results=20&start-index=1&min-viewability=none 

******************************************************************************** 

****************************** Amazon.com (1, 3, 5) ****************************** 
Found 0 results 
Downloading was aborted 
User-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.190 Safari/537.36
Server: auto
Making google query: https://www.google.com/search?q=%289798888438329+or+979-8888438329%29+site%3Awww.amazon.com
Failed to find any results on results page, with title: (9798888438329 or 979-8888438329) site:www.amazon.com - Google Search
No search engine results for terms: (9798888438329 OR 979-8888438329)
Trying the bing search engine instead
Making bing query: https://www.bing.com/search?q=%289798888438329+or+979-8888438329%29+site%3Awww.amazon.com
Failed to find any results on results page, with title: (9798888438329 or 979-8888438329) site:www.amazon.com - Search
No search engine results for terms: (9798888438329 OR 979-8888438329)
No matches found with identifiers, retrying using only title and authors. Query: ['(9798888438329', 'OR', '979-8888438329)']
User-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.87 Safari/537.36
Server: auto
Making google query: https://www.google.com/search?q=Irina%3A+Vampire+Cosmonaut+Vol.+5+Keisuke+Makino+site%3Awww.amazon.com 

******************************************************************************** 
The identify phase took 90.66 seconds 
The longest time (-1.000000) was taken by: Amazon.com 
Merging results from different sources 
We have 1 merged results, merging took: 0.00 seconds

And here is the log from Windows:

Running identify query with parameters: 
{'title': 'Irina: The Vampire Cosmonaut Vol. 5', 'authors': ['Keisuke Makino'], 'identifiers': {'isbn': '9798888438329'}, 'timeout': 30} 
Using plugins: Goodreads (1, 7, 9), Google (1, 1, 1), Amazon.com (1, 3, 5) 
The log from individual plugins is below 

****************************** Goodreads (1, 7, 9) ****************************** 
Found 1 results 
Downloading from Goodreads took 1.9767067432403564 


--- 
Title               : Irina: The Vampire Cosmonaut (Light Novel) Vol. 5
Author(s)           : Keisuke Makino & Karei
Publisher           : Airship
Series              : Irina: The Vampire Cosmonaut Light Novel #5
Languages           : eng
Rating              : 2
Published           : 2019-08-21T07:00:00+00:00
Identifiers         : goodreads:175423772, isbn:9798888438329
Comments            : 
Lev Leps, now vice-director of LAIKA44's training center, is raising the next generation of cosmonauts alongside vampire instructor Irina Luminesk. They soon discover that the engagement between fellow cosmonauts Mikhail and Roza is a farce concocted by the government to smother news of the space development standstill! Enraged at being pawns in a game of national pride, Lev and Irina struggle to move forward. A tragedy looming in the not-so-distant future might just push them over the brink?
 
identify - start. title=Irina: The Vampire Cosmonaut Vol. 5, authors=['Keisuke Makino'], identifiers={'isbn': '9798888438329'}
get_goodreads_id_from_identifiers - isbn=9798888438329
get_goodreads_id_using_api - identifiers=9798888438329
Querying using autocomplete API: https://www.goodreads.com/book/auto_complete?format=json&q=9798888438329
JSON Result: b'[{"imageUrl":"https://i.gr-assets.com/images/S/compressed.photo.goodreads.com/books/1686212853i/175423772._SX50_.jpg","bookId":"175423772","workId":"76761931","bookUrl":"/book/show/175423772-irina","from_search":true,"from_srp":true,"qid":"rBn4krYehN","rank":1,"title":"Irina: The Vampire Cosmonaut (Light Novel) Vol. 5","bookTitleBare":"Irina: The Vampire Cosmonaut (Light Novel) Vol. 5","numPages":203,"avgRating":"4.00","ratingsCount":1,"author":{"id":20098676,"name":"Keisuke Makino","isGoodreadsAuthor":false,"profileUrl":"https://www.goodreads.com/author/show/20098676.Keisuke_Makino","worksListUrl":"https://www.goodreads.com/author/list/20098676.Keisuke_Makino"},"kcrPreviewUrl":null,"description":{"html":"Lev Leps, now vice-director of LAIKA44\\u0026apos;s training center, is raising the next generation of cosmonauts alongside vampire instructor Irina Luminesk. They soon discover that the\xe2\x80\xa6","truncated":true,"fullContentUrl":"https://www.goodreads.com/book/show/175423772-irina"}}]'
Result using autocomplete API: 175423772
Get details attempt #1
Goodreads book url: 'https://www.goodreads.com/book/show/175423772'
Trying to parse book json for 2022 web page format
Json script node found, page in 2022+ html format
parse_title:  Irina: The Vampire Cosmonaut (Light Novel) Vol. 5
parse_authors - author=Keisuke Makino type=Contributor
parse_authors - author=KAREI type=Contributor
parse_details - goodreads_id: 175423772, mi: Title               : Irina: The Vampire Cosmonaut (Light Novel) Vol. 5
Author(s)           : Keisuke Makino & KAREI
parse_series: series_name='Irina: The Vampire Cosmonaut Light Novel', series_index='5.0'
parse_isbn:  9798888438329
parse_asin:  None
parse_rating:  4.0
parse_comments:  
Lev Leps, now vice-director of LAIKA44's training center, is raising the next generation of cosmonauts alongside vampire instructor Irina Luminesk. They soon discover that the engagement between fellow cosmonauts Mikhail and Roza is a farce concocted by the government to smother news of the space development standstill! Enraged at being pawns in a game of national pride, Lev and Irina struggle to move forward. A tragedy looming in the not-so-distant future might just push them over the brink?

Visiting image url: https://images-na.ssl-images-amazon.com/images/S/compressed.photo.goodreads.com/books/1686212853i/175423772.jpg
parse_tags: 
parse_publisher:  Airship
parse_publish_date: 2019-08-21 07:00:00+00:00 (First published)
parse_language:  eng 

******************************************************************************** 

****************************** Google (1, 1, 1) ****************************** 
Found 1 results 
Downloading from Google took 0.7018847465515137 


--- 
Title               : Irina: The Vampire Cosmonaut (Light Novel) Vol. 5
Author(s)           : Keisuke Makino & Karei
Publisher           : Seven Seas Entertainment
Tags                : Comics & Graphic Novels, Manga, Light Novel, Fiction, Science Fiction, Military, Alternative History
Languages           : eng
Published           : 2023-06-08T20:26:31.334970+00:00
Identifiers         : google:Gc6_EAAAQBAJ, isbn:9798888438329
Comments            : 
Lev Leps, now vice-director of LAIKA44's training center, is raising the next generation of cosmonauts alongside vampire instructor Irina Luminesk. They soon discover that the engagement between fellow cosmonauts Mikhail and Roza is a farce concocted by the government to smother news of the space development standstill! Enraged at being pawns in a game of national pride, Lev and Irina struggle to move forward. A tragedy looming in the not-so-distant future might just push them over the brink...
 
Making query: https://books.google.com/books/feeds/volumes?q=isbn%3A9798888438329&max-results=20&start-index=1&min-viewability=none 

******************************************************************************** 

****************************** Amazon.com (1, 3, 5) ****************************** 
Found 1 results 
Downloading from Amazon.com took 6.312291860580444 


--- 
Title               : Irina: The Vampire Cosmonaut  Vol. 5
Author(s)           : Keisuke Makino
Publisher           : Airship
Languages           : eng
Published           : 2023-07-04T00:00:00+00:00
Identifiers         : amazon:1685796516, isbn:9781685796518
Comments            : 
TERRESTRIAL TURMOIL  
Lev Leps, now vice-director of LAIKA44?s training center, is raising the next generation of cosmonauts alongside vampire instructor Irina Luminesk. They soon discover that the engagement between fellow cosmonauts Mikhail and Roza is a farce concocted by the government to smother news of the space development standstill! Enraged at being pawns in a game of national pride, Lev and Irina struggle to move forward. A tragedy looming in the not-so-distant future might just push them over the brink?
About the Author
Keisuke Makino is a writer in Japan best known for novels, screenplays, and video games, including Irina: The Vampire Cosmonaut , Persona 5 , and more.  
KAREI is an artist in Japan best known for illustrating light novel series such as Irina: The Vampire Cosmonaut (Light Novel).
 
User-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.87 Safari/537.36
Server: auto
Making google query: https://www.google.com/search?q=%289798888438329+or+979-8888438329%29+site%3Awww.amazon.com
Failed to find any results on results page, with title: (9798888438329 or 979-8888438329) site:www.amazon.com - Google Search
No search engine results for terms: (9798888438329 OR 979-8888438329)
Trying the bing search engine instead
Making bing query: https://www.bing.com/search?q=%289798888438329+or+979-8888438329%29+site%3Awww.amazon.com
Failed to find any results on results page, with title: (9798888438329 or 979-8888438329) site:www.amazon.com - Search
No search engine results for terms: (9798888438329 OR 979-8888438329)
No matches found with identifiers, retrying using only title and authors. Query: ['(9798888438329', 'OR', '979-8888438329)']
User-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36
Server: auto
Making google query: https://www.google.com/search?q=Irina%3A+Vampire+Cosmonaut+Vol.+5+Keisuke+Makino+site%3Awww.amazon.com
Ignoring 'Images for Irina: Vampire Cosmonaut Vol. 5 Keisuke Makino site:www.amazon.com' as it has no cached page
Ignoring 'Images for Irina: Vampire Cosmonaut Vol. 5 Keisuke Makino site:www.amazon.com' as it has no cached page
Ignoring 'Images for Irina: Vampire Cosmonaut Vol. 5 Keisuke Makino site:www.amazon.com' as it has no cached page
Ignoring 'Images for Irina: Vampire Cosmonaut Vol. 5 Keisuke Makino site:www.amazon.com' as it has no cached page
Ignoring 'Images for Irina: Vampire Cosmonaut Vol. 5 Keisuke Makino site:www.amazon.com' as it has no cached page
Ignoring div with no main result link
Downloading details from: https://webcache.googleusercontent.com/search?q=cache:utzqGHPndukJ:https://www.amazon.com/Irina-Vampire-Cosmonaut-Light-Novel/dp/1685796516&cd=9&hl=en&ct=clnk&gl=uk
Found image: https://m.media-amazon.com/images/I/81Liz3eBFOL._AC_UF350,350_QL50_.jpg
Downloading details from: https://webcache.googleusercontent.com/search?q=cache:NNJ0PyhgSfcJ:https://www.amazon.com/Irina-Vampire-Cosmonaut-Light-Novel-ebook/dp/B0C9LKS3XV&cd=10&hl=en&ct=clnk&gl=uk
Failed to find product description for url: 'https://webcache.googleusercontent.com/search?q=cache:NNJ0PyhgSfcJ:https://www.amazon.com/Irina-Vampire-Cosmonaut-Light-Novel-ebook/dp/B0C9LKS3XV&cd=10&hl=en&ct=clnk&gl=uk'
Downloading details from: https://webcache.googleusercontent.com/search?q=cache:_3IrzeiY_ZkJ:https://www.amazon.com/dp/B0B39KWN9N%3Fbinding%3Dpaperback%26searchxofy%3Dtrue%26ref_%3Ddbs_s_bs_series_rwt_tpbk%26sr%3D1-12&cd=11&hl=en&ct=clnk&gl=uk
Error parsing title for url: 'https://webcache.googleusercontent.com/search?q=cache:_3IrzeiY_ZkJ:https://www.amazon.com/dp/B0B39KWN9N%3Fbinding%3Dpaperback%26searchxofy%3Dtrue%26ref_%3Ddbs_s_bs_series_rwt_tpbk%26sr%3D1-12&cd=11&hl=en&ct=clnk&gl=uk'
Traceback (most recent call last):
  File "calibre\ebooks\metadata\sources\amazon.py", line 441, in parse_details
  File "calibre\ebooks\metadata\sources\amazon.py", line 592, in parse_title
ValueError: No title block found

Could not find title/authors/asin for 'https://webcache.googleusercontent.com/search?q=cache:_3IrzeiY_ZkJ:https://www.amazon.com/dp/B0B39KWN9N%3Fbinding%3Dpaperback%26searchxofy%3Dtrue%26ref_%3Ddbs_s_bs_series_rwt_tpbk%26sr%3D1-12&cd=11&hl=en&ct=clnk&gl=uk'
ASIN: 'B0B39KWN9N' Title: None Authors: [] 

******************************************************************************** 
The identify phase took 6.41 seconds 
The longest time (6.312292) was taken by: Amazon.com 
Merging results from different sources 
We have 2 merged results, merging took: 0.00 seconds

Does anyone have suggestions on how to fix this or at least a way to get more detailed logs on how long each request took?

Thanks

[kovidgoyal]

There is no OS specific code in metadata download. This will be some network issue on your linux machine.

[Quoth]

Sometimes, especially if the ISP doesn't support it, disabling IPv6 support in each network connection is needed. Any new connection will automatically have IPv6 enabled.

It could also be an issue on your Router or ISP. I also had to try a couple of models of USB-Ethernet dongles on my new Lenovo laptop as it has no Ethernet port and WiFi is too variable in performance.

Linux Mint 21.1 with Mate Desktop. I see 21.2 LTS Victoria is just released, but I'll test it on an old laptop.