[Metadata Source Plugin] Kobo Metadata

[Terisa de morgan]

Originally Posted by JSWolf

I've edited Kobo Metadata to fix the \d warning when running other plugins and looking at the log file. I've upped the version to 1.13.2.

Thank you, as I sometimes run calibre from a command windows this was bothering me.

[JSWolf]

Originally Posted by DNSB

I have done much the same with some of the plugins I use. However, the important point to me is that warning would not have caused the permissions issue and so your post was not going to be helpful to the OP as the attached file did not address their issue.

I wasn't posting to fix a permission error. Besides, I don't think the problem is with the plugin. I posted so if anyone else was looking as the log, they would not have the warning shown.

[JSWolf]

Originally Posted by Terisa de morgan

Thank you, as I sometimes run calibre from a command windows this was bothering me.

You're welcome. It was bothering me what I was helping someone who was having a problem with a Modify ePub problem. I didn't want to see the warning.

One thing I don't get is why anyone would complain about having the warning fixed.

[DNSB]

Originally Posted by JSWolf

One thing I don't get is why anyone would complain about having the warning fixed.

Perhaps that the poster immediately above your message was asking about an issue with that caused installing the Kobo Metadata plugin to fail? I suspect that many people would think that you were posting a version that fixed that install issue when you were correcting a cosmetic error that would not prevent the plugin from installing.

[JSWolf]

Originally Posted by DNSB

Perhaps that the poster immediately above your message was asking about an issue with that caused installing the Kobo Metadata plugin to fail? I suspect that many people would think that you were posting a version that fixed that install issue when you were correcting a cosmetic error that would not prevent the plugin from installing.

That is a permission error. That should have nothing to do with Kobo Metadata. If I had fixed something else, I would have mentioned it.

[DNSB]

Has anyone else tried adding the 'uses_camoufox_worker = True' bits and bobs to this or other plugins?

[JSWolf]

Originally Posted by DNSB

Has anyone else tried adding the 'uses_camoufox_worker = True' bits and bobs to this or other plugins?

Where does it go in a plugin?

[dunhill]

Following up on DNSB's question about uses_camoufox_worker: on its own that line does nothing for this plugin. The shared Camoufox worker in 9.17 only has handlers for Amazon and Google Images, and the plugin keeps fetching with cloudscraper anyway.

There is another way, though. calibre 9.17 also ships calibre.web.automate.recipes.Browser, the Camoufox browser used by news recipes. It doesn't need a handler: you create it and call open(url), much like mechanize. I've made a test build of Kobo Metadata (1.13.3, attached) that uses it directly.

Changes

• Kobo pages are loaded first with Camoufox, which runs Cloudflare's JavaScript like a real browser. If it still gets the Cloudflare challenge, it retries once.
• Fallback: on calibre older than 9.17, or if Camoufox fails, the plugin uses cloudscraper as before. Nothing breaks on older versions.
• Covers are fetched with a plain request first, since Kobo's CDN doesn't usually block. Camoufox is used only if that fails.
• New option in the plugin settings: "Use calibre's Camoufox browser (calibre 9.17+)". It's on by default and can be turned off.
• uses_camoufox_worker is now dynamic, True only when the browser will actually be used. calibre then waits up to 120 s for Kobo, since starting the browser is slow.
• The \d warning is fixed too (same fix as JSWolf's 1.13.2).

Limitations

• The first search is slow. calibre downloads Camoufox (a few hundred MB) and starts it, which can take a minute or two. Later searches in the same job are much faster.
• There may be two browsers running. If Amazon is also enabled, each source uses its own Camoufox instance, because the shared worker can't serve Kobo.
• Bulk jobs may only partly succeed. If Cloudflare gets more aggressive it can still show the challenge, and the plugin then falls back to cloudscraper.

How to test

1. Install the zip (Preferences > Plugins > Load plugin from file) and restart calibre.
2. In Preferences > Metadata download, temporarily leave only Kobo Metadata ticked.
3. Download metadata for a few books, including ones that used to fail. Be patient with the first one.
4. Click "View log" in the results window. The line "KoboMetadata: using calibre's Camoufox browser" confirms it's active. If something fails, please post the log in spoiler tags.

It would help to know whether it now finds books that used to fail, how bulk downloads behave, and which country/language store you use.

This is a test build, not an official release. @NotSimone, if it works well I'm happy to send it as a pull request.

@NotSimone, I've also opened an issue on GitHub with the source changes as a patch, so you can review exactly what was changed in the test build I posted above:
https://github.com/NotSimone/Kobo-Metadata/issues/19

It only touches __init__.py and kobo_metadata.py. It's your plugin, so feel free to take it, change it or drop it. Anyone testing the build, please keep posting results here.

[DNSB]

I'll give that a try. I was working on modifying the Camoufox worker to allow me to use other plugins with it but so far the results were less than satisfying.

[Terisa de morgan]

I will check because, as I've got my own version of some plugins, perhaps it's useful. Just a piece of warning: for me, not only the first download is slower.

[NotSimone]

Originally Posted by dunhill

@Terryw539, before reinstalling anything, please check this. The error is raised inside urllib3’s create_urllib3_context(), which opens the file named in the SSLKEYLOGFILE environment variable. On your PC that variable points to a file that can’t be written (virtual_file.log). Some programs set it automatically. calibre itself doesn’t use urllib3, which is why calibre works but this plugin fails.

To check, open a Command Prompt and run:

echo %SSLKEYLOGFILE%

If it prints a path (anything other than %SSLKEYLOGFILE%), that’s the cause. To fix it:

Open Control Panel → System → Advanced system settings → Environment Variables.
Delete SSLKEYLOGFILE, checking both User and System variables.
Close calibre and delete AppData\Roaming\calibre\plugins\Ko bo Metadata.zip (note the odd space in the name, probably a leftover).
Reinstall the plugin.

If you’d like to test a build with the fix included instead, let me know and I’ll send it to you.

@NotSimone, I opened an issue on GitHub with the diagnosis and a proposed patch: https://github.com/NotSimone/Kobo-Metadata/issues/18

The patch ignores SSLKEYLOGFILE when its path isn’t writable. It also loads the network libraries on first use instead of in the constructor, so an error like this goes to the metadata log instead of uninstalling the plugin.

It seems to me that if someone has this environmental variable set, we should probably use it instead of trying to automagically drop it. And if they've set it somewhere bad, its their problem. I'm a bit disinclined to add your fix unless this turns out to be a common problem or Calibre itself is adding borked env variables.

[NotSimone]

Originally Posted by JSWolf

You're welcome. It was bothering me what I was helping someone who was having a problem with a Modify ePub problem. I didn't want to see the warning.

One thing I don't get is why anyone would complain about having the warning fixed.

Sorry maybe I should clarify but I was asking about what you changed to fix this.

[dunhill]

Originally Posted by NotSimone

It seems to me that if someone has this environmental variable set, we should probably use it instead of trying to automagically drop it. And if they've set it somewhere bad, its their problem. I'm a bit disinclined to add your fix unless this turns out to be a common problem or Calibre itself is adding borked env variables.

Fair enough, that makes sense. It's a misconfiguration on that PC, not a plugin bug, so I agree there's no need to override the variable.

Thanks for taking a look.

[DNSB]

Originally Posted by NotSimone

Sorry maybe I should clarify but I was asking about what you changed to fix this.

Several ways of handling this. You can change a string to a raw string ( r"C:\Data\Test\" ), change the \ to \\ to escape the \ ( "C:\\Data\\Test\\" ) or you can try changing the backslash to a front slash ( r"C:/Data/Test/" ).