[Metadata Source Plugin] Deutsche Nationalbibliothek - Page 3

citronalco · 01-13-2022, 10:47 AM

OK, I've released v3.1.5

Moonbase59 · 01-15-2022, 06:06 AM

Thanks! For those of us who have tested using your zip: Should we force an upgrade or is it still the same?

citronalco · 01-16-2022, 04:55 AM

The release version is exactly the same as the test version, no update required.

Moonbase59 · 01-16-2022, 11:28 AM

Thanks for clarifying! Good job!

Wiggo · 01-19-2022, 11:39 AM

Hi citronalco,

I am not sure if this could have something to do with your plugin.

Problems with Umlaute in authors

citronalco · 01-20-2022, 05:21 PM

Quote:

Originally Posted by Wiggo

Hi citronalco,

I am not sure if this could have something to do with your plugin.

Problems with Umlaute in authors

I'm pretty sure it has.
Up to now I was totally unaware that there a several possible UTF-8 encodings for the same (visual) character. Seems like one can either simply use an "ö" character ("normalized form") like everyone does, or use "o" and put a two-dots-on-top character over it ("composed form"). I checked, and of course DNB delivers its data in the composed form.

kovidgoyal wrote "calibre usually normalizes metadata it reads from most sources", which probably means Calibre plugins like the DNB plugin should normalize those composed UTF-8 texts. So I'm going to do that.

I'm somewhat busy ATM, so this could take a while. Btw: I happily accept pull requests

Wiggo · 01-21-2022, 12:35 AM

I am glad to have found the problem.

Take your time and thank you in advance.

Moonbase59 · 01-21-2022, 04:19 AM

If really the plugin has to do it (vs. "strings returned by a plugin are normalized by Calibre"), be aware that there are some caveats lurking here:

You’ll probably want use the NFC (Normalization Form Canonical Composition), as in unicodedata.normalize('NFC', string).
Personally, I’d advise against using NFKC (the "compatibility" composition), because it breaks too much, like super- and subscripts, Roman numerals. Even the angstrom sign (Å) will silently be converted to the Swedish letter (Å).
Be sure the string is complete, i.e. not starting with a combining character or ending with a joining one.
Some more info on Wikipedia and unicode.org.

citronalco · 01-24-2022, 12:44 PM

Please try out this the attached test version.
The only change: The metadata DNB sends gets converted to NFC before any processing.

Wiggo · 01-25-2022, 04:11 AM

Works fine!

Just did some tests on books with ö, ä, ü, á, è, é and í.

Maybe Moonbase could do some additional testing?

Thank you so much!

Moonbase59 · 01-25-2022, 09:12 AM

So you’re normalizing the whole XML using Calibre’s "normalize()". Probably a good idea, in case something changes within Calibre. Looks ok to me, from the code standpoint, but I hadn’t yet time to check much.

Problem being that I mainly read English, thus having not so many books with diacritics. Maybe I can find and test some German books later, or create some dummies for testing.

Thanks for caring @citronalco!

EDIT: @Wiggo: Tried some German books, all data gotten from DNB plugin was normalized (as far as I checked, mainly titles and authors). Even tested Andreas Schöfl’s »Altherrenjagd«. All good. Now for a Weißbier! Prost.

citronalco · 01-30-2022, 03:17 AM

Thank you all for testing! v3.1.6 with unicode in NFC is released.
It's identical to the test version in post #39.

Wiggo · 01-30-2022, 10:05 AM

Thanks for your work and time

citronalco · 03-31-2022, 02:45 PM

Released v3.1.7:
Some Python installations do not support DFN's CA, so some people could not get data from DNB at all. Works now.
No other changes.

Wiggo · 07-16-2022, 05:57 AM

It looks like it doesn't work with Calibre 6.

01-21-2022, 04:19 AM	#38
Moonbase59 Zealot Posts: 111 Karma: 196896 Join Date: Oct 2021 Location: Germany Device: Tolino Vision 5, Tolino Tab 8"	If really the plugin has to do it (vs. "strings returned by a plugin are normalized by Calibre"), be aware that there are some caveats lurking here: You’ll probably want use the NFC (Normalization Form Canonical Composition), as in unicodedata.normalize('NFC', string). Personally, I’d advise against using NFKC (the "compatibility" composition), because it breaks too much, like super- and subscripts, Roman numerals. Even the angstrom sign (Å) will silently be converted to the Swedish letter (Å). Be sure the string is complete, i.e. not starting with a combining character or ending with a joining one. Some more info on Wikipedia and unicode.org. Attached Thumbnails Last edited by Moonbase59; 01-21-2022 at 04:35 AM.

01-25-2022, 09:12 AM	#41
Moonbase59 Zealot Posts: 111 Karma: 196896 Join Date: Oct 2021 Location: Germany Device: Tolino Vision 5, Tolino Tab 8"	So you’re normalizing the whole XML using Calibre’s "normalize()". Probably a good idea, in case something changes within Calibre. Looks ok to me, from the code standpoint, but I hadn’t yet time to check much. Problem being that I mainly read English, thus having not so many books with diacritics. Maybe I can find and test some German books later, or create some dummies for testing. Thanks for caring @citronalco! EDIT: @Wiggo: Tried some German books, all data gotten from DNB plugin was normalized (as far as I checked, mainly titles and authors). Even tested Andreas Schöfl’s »Altherrenjagd«. All good. Now for a Weißbier! Prost. Last edited by Moonbase59; 01-25-2022 at 09:42 AM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
[Metadata Source Plugin] Comicvine	chewt0y	Plugins	88	07-11-2022 12:00 PM
[Metadata Source Plugin] INMONDADORI	Pr.BarnArt	Plugins	7	12-03-2021 12:56 PM
[Metadata Source Plugin] Amazon.CN	fated	Plugins	0	11-20-2014 04:59 PM
[Metadata Source Plugin] DIZZIE_NL	Pr.BarnArt	Plugins	7	08-11-2014 01:48 PM
[Metadata Source Plugin] catawiki	drMerry	Plugins	0	06-28-2011 08:42 PM

01-13-2022, 10:47 AM	#31
citronalco Enthusiast Posts: 32 Karma: 10 Join Date: Jul 2018 Device: PocketBook Touch HD	OK, I've released v3.1.5

01-15-2022, 06:06 AM	#32
Moonbase59 Zealot Posts: 111 Karma: 196896 Join Date: Oct 2021 Location: Germany Device: Tolino Vision 5, Tolino Tab 8"	Thanks! For those of us who have tested using your zip: Should we force an upgrade or is it still the same?

01-16-2022, 04:55 AM	#33
citronalco Enthusiast Posts: 32 Karma: 10 Join Date: Jul 2018 Device: PocketBook Touch HD	The release version is exactly the same as the test version, no update required.

01-16-2022, 11:28 AM	#34
Moonbase59 Zealot Posts: 111 Karma: 196896 Join Date: Oct 2021 Location: Germany Device: Tolino Vision 5, Tolino Tab 8"	Thanks for clarifying! Good job!

01-19-2022, 11:39 AM	#35
Wiggo Leftutti Posts: 400 Karma: 1121267 Join Date: Feb 2019 Location: Bavaria Device: iPad Pro, Kobo Libra 2	Hi citronalco, I am not sure if this could have something to do with your plugin. Problems with Umlaute in authors

01-21-2022, 12:35 AM	#37
Wiggo Leftutti Posts: 400 Karma: 1121267 Join Date: Feb 2019 Location: Bavaria Device: iPad Pro, Kobo Libra 2	I am glad to have found the problem. Take your time and thank you in advance.

01-25-2022, 04:11 AM	#40
Wiggo Leftutti Posts: 400 Karma: 1121267 Join Date: Feb 2019 Location: Bavaria Device: iPad Pro, Kobo Libra 2	Works fine! Just did some tests on books with ö, ä, ü, á, è, é and í. Maybe Moonbase could do some additional testing? Thank you so much!

01-30-2022, 03:17 AM	#42
citronalco Enthusiast Posts: 32 Karma: 10 Join Date: Jul 2018 Device: PocketBook Touch HD	Thank you all for testing! v3.1.6 with unicode in NFC is released. It's identical to the test version in post #39.

01-30-2022, 10:05 AM	#43
Wiggo Leftutti Posts: 400 Karma: 1121267 Join Date: Feb 2019 Location: Bavaria Device: iPad Pro, Kobo Libra 2	Thanks for your work and time

03-31-2022, 02:45 PM	#44
citronalco Enthusiast Posts: 32 Karma: 10 Join Date: Jul 2018 Device: PocketBook Touch HD	Released v3.1.7: Some Python installations do not support DFN's CA, so some people could not get data from DNB at all. Works now. No other changes.

Advert

Advert

07-16-2022, 05:57 AM	#45
Wiggo Leftutti Posts: 400 Karma: 1121267 Join Date: Feb 2019 Location: Bavaria Device: iPad Pro, Kobo Libra 2	It looks like it doesn't work with Calibre 6.