[Metadata Source Plugin] Kobo Metadata

[NotSimone]

Originally Posted by DNSB

Is this the language code entered on the line that reads: 2 Letter language code to search for? I tried 'all' in there and the generated URL started with https://www.kobo.com/ca/all/search? where all is where a 2 letter language code should be found. It's possible that Kobo updated their web server code to only allow valid language codes in that position.

What version were you using? jjangsangy actually made a change to this in version 1.12.0 that fixed the handling of "all".

Originally Posted by nephtys59

Something has changed. Kobo metadata plugin stopped working for me a couple days ago. I changed from "all" to "it" in settings and now it works but doesn't get ISBN and publication date like it did before. Probably the plugin needs an update.

Do you have a specific example? Kobo have in the past put different versions of the webpage for different countries, probably to do some A/B testing. I would suggest trying another country if it breaks.

Originally Posted by Cactus Chef

I think I have the same question as nAhu...is it possible to make the plug-in pull down bold text in the description when it grabs from Kobo? It pulls the description successfully, but it's all in plaintext, and I also notice sometimes line-breaks are missing. So I end up manually copy-pasting from the Kobo description page anyway just to make it look prettier.

The plugin doesn't strip any of the formatting and I have reasonably recently downloaded metadata with bolded text. Do you have a specific example of one where it is broken? If I were to hazard a wild guess, this could be related to which country you have picked.

[NotSimone]

More broadly, I don't really check this site and I somehow missed the email notifications. It would be better if any bug reports go on the github, or at the least, include specific examples of something that is broken.

[nephtys59]

Do you have a specific example? Kobo have in the past put different versions of the webpage for different countries, probably to do some A/B testing. I would suggest trying another country if it breaks.

Ok, I made a test with this book I purchased from Kobo italian site: https://www.kobo.com/it/it/ebook/all...gnor-weynfeldt

In January the "all" setting didn't work at all, so I switched it to "it" but the data "Publication date" and "ISBN" are not retrieved, even if they are present on the web page (bottom).

This morning, making tests for you, I switched back to "all" settings and magically started working again, retrieving also Publ. date and ISBN. Who knows what happened?

See images attached. Thanks!

[NotSimone]

Originally Posted by nephtys59

Ok, I made a test with this book I purchased from Kobo italian site: https://www.kobo.com/it/it/ebook/all...gnor-weynfeldt

In January the "all" setting didn't work at all, so I switched it to "it" but the data "Publication date" and "ISBN" are not retrieved, even if they are present on the web page (bottom).

This morning, making tests for you, I switched back to "all" settings and magically started working again, retrieving also Publ. date and ISBN. Who knows what happened?

See images attached. Thanks!

Hmm we'll probably chalk that up to Kobo doing some testing with ui updates then. Hopefully they don't do that too many more times

[nAhu]

Originally Posted by NotSimone

The plugin doesn't strip any of the formatting and I have reasonably recently downloaded metadata with bolded text. Do you have a specific example of one where it is broken? If I were to hazard a wild guess, this could be related to which country you have picked.

Hi NotSimone
Last version of the plugin.
Using with "French" store.

But format is not fetched neither isbn code.
Tryed with : https://www.kobo.com/fr/fr/ebook/la-...TcWdzwt&cPos=1

nothing bold in the result but displayed as bold on the site and isbn is here.

Log in spoiler tag.

Thx for the dev

Spoiler:

#################### Log for La Méthode Glucose Goddess - Le programme scientifique de 4 semaines pour réduire vos fringales, retrouver votre énergie et vous sentir en pleine forme ####################
Running identify query with parameters: 
{'title': 'La Méthode Glucose Goddess - Le programme scientifique de 4 semaines pour réduire vos fringales, retrouver votre énergie et vous sentir en pleine forme', 'authors': ['Jessie Inchauspé'], 'identifiers': {}, 'timeout': 30} 
Using plugins: Kobo Metadata (1, 12, 0) 
The log from individual plugins is below 

****************************** Kobo Metadata (1, 12, 0) ****************************** 
Found 1 results 
Downloading from Kobo Metadata took 1.6168560981750488 


--- 
Title               : La Méthode Glucose Goddess - Le programme scientifique de 4 semaines pour réduire vos fringales, retrouver votre énergie et vous sentir en pleine forme
Author(s)           : Jessie Inchauspé
Publisher           : GROUPE ROBERT LAFFONT
Tags                : Santé, Essais et documents, Santé et bien-être, Nutrition et régime alimentaire, Régimes
Comments            : 
            Fringales ? Fatigue chronique ? Accro au sucre ?
Problèmes hormonaux ? En mauvaise santé ?
Votre glycémie joue sûrement aux montagnes russes.
Ce livre est votre solution.
Dans son premier livre, Faites votre glucose révolution, Jessie Inchauspé présentait le rôle crucial du glucose dans l'organisme et livrait dix conseils scientifiques pour lisser votre glycémie et vous libérer de nombreux symptômes.
Avec La Méthode Glucose Goddess, elle présente son  programme de quatre semaines et en quatre étapes pour intégrer ces conseils facilement, ludiquement, et sans se priver.
Dans cet ouvrage tant attendu, vous trouverez plus de cent recettes, un  carnet de notes, ainsi que les témoignages et conseils de deux mille sept cents personnes qui ont testé le programme en avant-première.
Cette méthode agit de façon puissante sur la santé et vous aidera à retrouver une énergie sans limite, à dire adieu aux fringales, ralentir le vieillissement, combattre l'inflammation, retrouver un équilibre hormonal, dormir mieux que jamais, faire reculer le diabète et adopter des habitudes pour la vie. Sans compter les calories, et en mangeant tout ce que vous aimez.
Pour une lecture optimale, privilégier une lecture sur ordinateur ou sur tablette.

         
KoboMetadata::identify: title: La Méthode Glucose Goddess - Le programme scientifique de 4 semaines pour réduire vos fringales, retrouver votre énergie et vous sentir en pleine forme, authors: ['Jessie Inchauspé'], identifiers: {}
No matches found with identifiers, falling back to general search.
KoboMetadata::perform_search: Searching with query: La Méthode Glucose Goddess Le programme scientifique de 4 semaines pour réduire vos fringales, retrouver votre énergie et vous sentir en pleine forme Jessie Inchauspé
KoboMetadata::perform_query: Searching for book with url: https://www.kobo.com/fr/fr/search?qu...fclanguages=fr
KoboMetadata::parse_search_page: Detected new search page
KoboMetadata::fetch_metadata: Looking up metadata with url: https://www.kobo.com/fr/fr/ebook/la-...Ew8vYuk&cPos=1
KoboMetadata::parse_book_page: Got title: La Méthode Glucose Goddess - Le programme scientifique de 4 semaines pour réduire vos fringales, retrouver votre énergie et vous sentir en pleine forme
KoboMetadata::parse_book_page: Got authors: ['Jessie Inchauspé']
KoboMetadata::parse_book_page: Got publisher: GROUPE ROBERT LAFFONT
KoboMetadata::parse_book_page: Got tags: {'Santé', 'Essais et documents', 'Santé et bien-être', 'Nutrition et régime alimentaire', 'Régimes'}
KoboMetadata::parse_book_page: Got comments: 
            Fringales ? Fatigue chronique ? Accro au sucre ?
Problèmes hormonaux ? En mauvaise santé ?
Votre glycémie joue sûrement aux montagnes russes.
Ce livre est votre solution.
Dans son premier livre, Faites votre glucose révolution, Jessie Inchauspé présentait le rôle crucial du glucose dans l'organisme et livrait dix conseils scientifiques pour lisser votre glycémie et vous libérer de nombreux symptômes.
Avec La Méthode Glucose Goddess, elle présente son  programme de quatre semaines et en quatre étapes pour intégrer ces conseils facilement, ludiquement, et sans se priver.
Dans cet ouvrage tant attendu, vous trouverez plus de cent recettes, un  carnet de notes, ainsi que les témoignages et conseils de deux mille sept cents personnes qui ont testé le programme en avant-première.
Cette méthode agit de façon puissante sur la santé et vous aidera à retrouver une énergie sans limite, à dire adieu aux fringales, ralentir le vieillissement, combattre l'inflammation, retrouver un équilibre hormonal, dormir mieux que jamais, faire reculer le diabète et adopter des habitudes pour la vie. Sans compter les calories, et en mangeant tout ce que vous aimez.
Pour une lecture optimale, privilégier une lecture sur ordinateur ou sur tablette.

        
KoboMetadata::parse_book_page_for_cover: Got cover: https://cdn.kobo.com/book-images/353...eine-forme.jpg
Found 1 match(es) using general search. 

******************************************************************************** 
The identify phase took 1.81 seconds 
The longest time (1.616856) was taken by: Kobo Metadata 
Merging results from different sources 
We have 1 merged results, merging took: 0.00 seconds 

****************************** Kobo Metadata Covers ****************************** 
Request extra headers: [('User-agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/144.0.0.0 Safari/537.36 Edg/144.0.0.0')] 
Downloaded cover: 1400x1823 
Took 1.5192840099334717 seconds 
KoboMetadata::download_cover: No cached url found, running identify
KoboMetadata::get_cover_url: title: La Méthode Glucose Goddess - Le programme scientifique de 4 semaines pour réduire vos fringales, retrouver votre énergie et vous sentir en pleine forme, authors: ['Jessie Inchauspé'], identifiers: {}
KoboMetadata::get_cover_url:: No identifier - performing search
KoboMetadata::perform_search: Searching with query: La Méthode Glucose Goddess Le programme scientifique de 4 semaines pour réduire vos fringales, retrouver votre énergie et vous sentir en pleine forme Jessie Inchauspé
KoboMetadata::perform_query: Searching for book with url: https://www.kobo.com/fr/fr/search?qu...fclanguages=fr
KoboMetadata::parse_search_page: Detected new search page
KoboMetadata::parse_book_page_for_cover: Got cover: https://cdn.kobo.com/book-images/353...eine-forme.jpg 

********************************************************************************

[NotSimone]

I've updated the plugin with a fix to the text bolding - it should work across all countries so let me know if there is an instance where it is broken.

Thanks for the link nAhu, I've found the issue with broken tagging on non-english languages. Whether this is new I don't know, but the metadata fields at the bottom of the page are being translated into the page language, which breaks the text matching that I use. Fixing this isn't hard, but would be a laborious process of going to each language, finding the text, and adding a mapping in the plugin. Not hard, but time consuming and I don't have much time for the next few months at least. Anyone is welcome to raise a PR if they want to do this though

[nAhu]

Originally Posted by NotSimone

I've updated the plugin with a fix to the text bolding - it should work across all countries so let me know if there is an instance where it is broken.

Thx, tested just now, seems good for bold and italic

[Cactus Chef]

Trying to test the new version, but it seems to be unable to reach Kobo now due to Cloudflare.

I was trying to get metadata for this book that I just purchased:
https://www.kobo.com/us/en/ebook/the-good-old-stuff-4

calibre, version 9.4.0
ERROR: No matches found: <p>Failed to find any books that match your search. Try making the search <b>less specific</b>. For example, use only the author's last name and a single distinctive word from the title.<p>To see the full log, click "Show details".

Running identify query with parameters: 
{'title': 'The Good Old Stuff', 'authors': ['Gardner Dozois'], 'identifiers': {}, 'timeout': 30} 
Using plugins: Kobo Metadata (1, 12, 1) 
The log from individual plugins is below 

****************************** Kobo Metadata (1, 12, 1) ****************************** 
Found 0 results 
Downloading from Kobo Metadata took 30.89536142349243 
KoboMetadata::identify: title: The Good Old Stuff, authors: ['Gardner Dozois'], identifiers: {}
No matches found with identifiers, falling back to general search.
KoboMetadata::perform_search: Searching with query: The Good Old Stuff Gardner Dozois
KoboMetadata::perform_query: Searching for book with url: https://www.kobo.com/us/en/search?query=The+Good+Old+Stuff+Gardner+Dozois&fcmedia=Book&pageNumber=1&fclanguages=en
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=The+Good+Old+Stuff+Gardner+Dozois&fcmedia=Book&pageNumber=1&fclanguages=en
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=The+Good+Old+Stuff+Gardner+Dozois&fcmedia=Book&pageNumber=1&fclanguages=en
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=The+Good+Old+Stuff+Gardner+Dozois&fcmedia=Book&pageNumber=1&fclanguages=en
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=The+Good+Old+Stuff+Gardner+Dozois&fcmedia=Book&pageNumber=1&fclanguages=en
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=The+Good+Old+Stuff+Gardner+Dozois&fcmedia=Book&pageNumber=1&fclanguages=en
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=The+Good+Old+Stuff+Gardner+Dozois&fcmedia=Book&pageNumber=1&fclanguages=en
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=The+Good+Old+Stuff+Gardner+Dozois&fcmedia=Book&pageNumber=1&fclanguages=en
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=The+Good+Old+Stuff+Gardner+Dozois&fcmedia=Book&pageNumber=1&fclanguages=en
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=The+Good+Old+Stuff+Gardner+Dozois&fcmedia=Book&pageNumber=1&fclanguages=en
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=The+Good+Old+Stuff+Gardner+Dozois&fcmedia=Book&pageNumber=1&fclanguages=en
KoboMetadata::parse_search_page: Detected new search page
KoboMetadata::fetch_metadata: Looking up metadata with url: https://www.kobo.com/us/en/ebook/the-good-new-stuff-1?sId=f019167c-8e09-480b-9052-2724ec6bed0c&ssId=ivSwvoW7eDUFfUORNjKYb&cPos=1
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/ebook/the-good-new-stuff-1?sId=f019167c-8e09-480b-9052-2724ec6bed0c&ssId=ivSwvoW7eDUFfUORNjKYb&cPos=1
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/ebook/the-good-new-stuff-1?sId=f019167c-8e09-480b-9052-2724ec6bed0c&ssId=ivSwvoW7eDUFfUORNjKYb&cPos=1
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/ebook/the-good-new-stuff-1?sId=f019167c-8e09-480b-9052-2724ec6bed0c&ssId=ivSwvoW7eDUFfUORNjKYb&cPos=1
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/ebook/the-good-new-stuff-1?sId=f019167c-8e09-480b-9052-2724ec6bed0c&ssId=ivSwvoW7eDUFfUORNjKYb&cPos=1
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/ebook/the-good-new-stuff-1?sId=f019167c-8e09-480b-9052-2724ec6bed0c&ssId=ivSwvoW7eDUFfUORNjKYb&cPos=1
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/ebook/the-good-new-stuff-1?sId=f019167c-8e09-480b-9052-2724ec6bed0c&ssId=ivSwvoW7eDUFfUORNjKYb&cPos=1
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/ebook/the-good-new-stuff-1?sId=f019167c-8e09-480b-9052-2724ec6bed0c&ssId=ivSwvoW7eDUFfUORNjKYb&cPos=1
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/ebook/the-good-new-stuff-1?sId=f019167c-8e09-480b-9052-2724ec6bed0c&ssId=ivSwvoW7eDUFfUORNjKYb&cPos=1
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/ebook/the-good-new-stuff-1?sId=f019167c-8e09-480b-9052-2724ec6bed0c&ssId=ivSwvoW7eDUFfUORNjKYb&cPos=1
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/ebook/the-good-new-stuff-1?sId=f019167c-8e09-480b-9052-2724ec6bed0c&ssId=ivSwvoW7eDUFfUORNjKYb&cPos=1
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/ebook/the-good-new-stuff-1?sId=f019167c-8e09-480b-9052-2724ec6bed0c&ssId=ivSwvoW7eDUFfUORNjKYb&cPos=1
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/ebook/the-good-new-stuff-1?sId=f019167c-8e09-480b-9052-2724ec6bed0c&ssId=ivSwvoW7eDUFfUORNjKYb&cPos=1
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/ebook/the-good-new-stuff-1?sId=f019167c-8e09-480b-9052-2724ec6bed0c&ssId=ivSwvoW7eDUFfUORNjKYb&cPos=1
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/ebook/the-good-new-stuff-1?sId=f019167c-8e09-480b-9052-2724ec6bed0c&ssId=ivSwvoW7eDUFfUORNjKYb&cPos=1
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/ebook/the-good-new-stuff-1?sId=f019167c-8e09-480b-9052-2724ec6bed0c&ssId=ivSwvoW7eDUFfUORNjKYb&cPos=1
KoboMetadata::get_webpage: Could not defeat cloudflare protection - giving up for https://www.kobo.com/us/en/ebook/the-good-new-stuff-1?sId=f019167c-8e09-480b-9052-2724ec6bed0c&ssId=ivSwvoW7eDUFfUORNjKYb&cPos=1
KoboMetadata::fetch_metadata: Could not get url: https://www.kobo.com/us/en/ebook/the-good-new-stuff-1?sId=f019167c-8e09-480b-9052-2724ec6bed0c&ssId=ivSwvoW7eDUFfUORNjKYb&cPos=1 

******************************************************************************** 
The identify phase took 31.04 seconds 
The longest time (30.895361) was taken by: Kobo Metadata 
Merging results from different sources 
We have 0 merged results, merging took: 0.00 seconds