[Metadata Source Plugin] Kobo Metadata

[NotSimone]

I've noticed there were a few issues fetching metadata a few days ago, but it seems alright now. I'm guessing they're fiddling around with the pages (again).

[ownedbycats]

Originally Posted by NotSimone

I've noticed there were a few issues fetching metadata a few days ago, but it seems alright now. I'm guessing they're fiddling around with the pages (again).

Maybe it's worth mentioning that the Cloudflare prompt's been popping up for me on the site a bit more over the past week or so.

[josephyw]

Originally Posted by JSWolf

Kobo Metadata is not working. In the log, in from of the enable JavaScript, lines is a Rakuten Kobo graphic logo.

****************************** Kobo Metadata Covers ****************************** 
Request extra headers: [('User-agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:130.0) Gecko/20100101 Firefox/130.0')] 
Failed to download valid cover 
Took 0.9577634334564209 seconds 
KoboMetadata::download_cover: No cached url found, running identify
KoboMetadata::get_cover_url: title: Mary, Mary, authors: ['James Patterson'], identifiers: {}
KoboMetadata::get_cover_url:: No identifier - performing search
KoboMetadata::perform_search: Searching with query: Mary, Mary James Patterson
KoboMetadata::perform_query: Searching for book with url: https://www.kobo.com/us/en/search?query=Mary%2C+Mary+James+Patterson&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::parse_search_page: Found no matches or bad page


Enable JavaScript and cookies to continue

KoboMetadata::parse_search_page: Found no matches or bad page


Enable JavaScript and cookies to continue

KoboMetadata::parse_search_page: Found no matches or bad page


Enable JavaScript and cookies to continue

KoboMetadata::get_cover_url:: No search results
KoboMetadata::download_cover: Could not get cover 

********************************************************************************

I am suddenly having this same issue. I am attaching a screenshot of my log, so you can see what the logo and Javascript message look like in context.

[JSWolf]

All I can say is that Cloudflare can bugger off.

[NotSimone]

I've just added a check for the new cloudflare fail page - hopefully this sorts it out (for now)

[josephyw]

Originally Posted by NotSimone

I've just added a check for the new cloudflare fail page - hopefully this sorts it out (for now)

I haven't had an issue since the update. Thank you!

[josephyw]

I may have spoken too soon, as I am getting severely rate-limited by Cloudflare and can't typically look up more than one or two books before I run into these new errors:

Running identify query with parameters: 
{'title': 'Woman of Interest', 'authors': ["Tracy O'Neill"], 'identifiers': {'isbn': '9780063309883', 'google': 'MhXZEAAAQBAJ'}, 'timeout': 30} 
Using plugins: Kobo Metadata (1, 9, 1), Google (1, 1, 1), Amazon.com (1, 3, 11) 
The log from individual plugins is below 
Not waiting any longer for more results. Still running sources: 
	Kobo Metadata 

****************************** Kobo Metadata (1, 9, 1) ****************************** 
Found 0 results 
Downloading was aborted 
KoboMetadata::identify: title: Woman of Interest, authors: ["Tracy O'Neill"], identifiers: {'isbn': '9780063309883', 'google': 'MhXZEAAAQBAJ'}
KoboMetadata::perform_isbn_search: Getting metadata with isbn: 9780063309883
KoboMetadata::perform_query: Searching for book with url: https://www.kobo.com/us/en/search?query=9780063309883&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9780063309883&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9780063309883&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9780063309883&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9780063309883&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9780063309883&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9780063309883&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9780063309883&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9780063309883&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9780063309883&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9780063309883&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9780063309883&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9780063309883&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9780063309883&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9780063309883&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9780063309883&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - giving up for https://www.kobo.com/us/en/search?query=9780063309883&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::perform_query: Could not get url: https://www.kobo.com/us/en/search?query=9780063309883&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::perform_search: Searching with query: Woman of Interest Tracy O'Neill
KoboMetadata::perform_query: Searching for book with url: https://www.kobo.com/us/en/search?query=Woman+of+Interest+Tracy+O%27Neill&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=Woman+of+Interest+Tracy+O%27Neill&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=Woman+of+Interest+Tracy+O%27Neill&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=Woman+of+Interest+Tracy+O%27Neill&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=Woman+of+Interest+Tracy+O%27Neill&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=Woman+of+Interest+Tracy+O%27Neill&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=Woman+of+Interest+Tracy+O%27Neill&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=Woman+of+Interest+Tracy+O%27Neill&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=Woman+of+Interest+Tracy+O%27Neill&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=Woman+of+Interest+Tracy+O%27Neill&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=Woman+of+Interest+Tracy+O%27Neill&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=Woman+of+Interest+Tracy+O%27Neill&fcmedia=Book&pageNumber=1&fclanguages=all 

********************************************************************************

[NotSimone]

Yeah I ran into this myself just then but it cleared up after half an hour ish. I suspect that stashing the cookies might help - I'll take a look at it some time next week.

[ownedbycats]

I wonder if visiting the site in a browser before attempting a download might help?

[NotSimone]

Thanks to jjangsangy for adding the capability to link to Kobo from the old Kobo Books kobo identifier!

Note that this will not write new Kobo Books identifiers, the plugin will continue to only write isbn identifiers. I also fixed a bug that was stopping the isbn identifiers from working properly - they should link through now.

I wonder if visiting the site in a browser before attempting a download might help?

Thats a solid maybe. From what I can tell, Cloudflare is running some kind of ML algorithm that looks for suspicious traffic, so having "good" traffic from the same IP may help. But it is likely insufficient on its own - you won't have matching browser user agents or cookies.

Actually, Cloudflare adjusted the way their cookies worked last week so my work-in-progress workaround stopped working too
Maybe the way to go is to add an option for users to link to a flaresolvarr server - I'll think about it.

[JSWolf]

I went to that URL in Firefox and the next cover search works. But if I then do yet another cover search, it fails.

****************************** Kobo Metadata Covers ****************************** 
Request extra headers: [('User-agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36 Edg/131.0.0.0')] 
Failed to download valid cover 
Took 18.18557119369507 seconds 
KoboMetadata::download_cover: No cached url found, running identify
KoboMetadata::get_cover_url: title: Drive, authors: ['James S. A. Corey'], identifiers: {}
KoboMetadata::get_cover_url:: No identifier - performing search
KoboMetadata::perform_search: Searching with query: Drive James Corey
KoboMetadata::perform_query: Searching for book with url: https://www.kobo.com/us/all/search?query=Drive+James+Corey&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/all/search?query=Drive+James+Corey&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/all/search?query=Drive+James+Corey&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/all/search?query=Drive+James+Corey&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/all/search?query=Drive+James+Corey&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/all/search?query=Drive+James+Corey&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/all/search?query=Drive+James+Corey&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/all/search?query=Drive+James+Corey&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/all/search?query=Drive+James+Corey&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/all/search?query=Drive+James+Corey&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/all/search?query=Drive+James+Corey&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/all/search?query=Drive+James+Corey&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/all/search?query=Drive+James+Corey&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/all/search?query=Drive+James+Corey&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/all/search?query=Drive+James+Corey&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/all/search?query=Drive+James+Corey&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - giving up for https://www.kobo.com/us/all/search?query=Drive+James+Corey&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::perform_query: Could not get url: https://www.kobo.com/us/all/search?query=Drive+James+Corey&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_cover_url:: No search results
KoboMetadata::download_cover: Could not get cover

[JSWolf]

@NotSimone I have an idea that might work to fix the cover problem.

When plugin fails to get the cover, send the URL out to the system so it views the URL in the default browser. Then have it start over getting the cover. I think the delay between sending the URL and starting over should be 1 minute.

[NotSimone]

Originally Posted by JSWolf

@NotSimone I have an idea that might work to fix the cover problem.

When plugin fails to get the cover, send the URL out to the system so it views the URL in the default browser. Then have it start over getting the cover. I think the delay between sending the URL and starting over should be 1 minute.

That is an interesting thought and from a very quick test, it looks possible to open a browser page from within the plugin. But there is a caveat - in your log, we couldn't beat the cloudflare protection while performing the search. Notably, we never got to the actual book page, so we have no idea where the cover is going to be.

Imo, the way to do this properly is to interface with a cloudflare defeating app like flare-bypasser or FlareSolverr, but both of these are not particularly easy to set up for most people so I'm not sure there is a whole lot of value in doing this, especially when the current set up already mostly works.

[DNSB]

For what it may be worth, I tried using FlareSolver with the FanFicFare plugin when I was experiencing too many failures. It never seemed to work for long and became a game of whack-a-mole. Wait for new version, install it, works for a while, stops working, go back to waiting for new version. I just use the browser cache now. A pain but it works fairly reliably.

[JSWolf]

Originally Posted by NotSimone

That is an interesting thought and from a very quick test, it looks possible to open a browser page from within the plugin. But there is a caveat - in your log, we couldn't beat the cloudflare protection while performing the search. Notably, we never got to the actual book page, so we have no idea where the cover is going to be.

Imo, the way to do this properly is to interface with a cloudflare defeating app like flare-bypasser or FlareSolverr, but both of these are not particularly easy to set up for most people so I'm not sure there is a whole lot of value in doing this, especially when the current set up already mostly works.

I do the cover search and it fails. I can do it again and it will still fail. I go to the log and get the URL. I put it in Firefox and go there. Then I do another cover search and it works. Another cover search after that fails.

So if Firefox goes to the URL that fails and then the plugin runs again, it should work. Firefox is my default browser.

I'd be happy to beta test a new version that does it the way I suggested.