[Metadata Source Plugin] Deutsche Nationalbibliothek

[gr00vian]

I also have the problem mentioned by Wiggo.

@citronalco:

Error message:

Plugin DNB_DE failed
Traceback (most recent call last):
File "calibre_plugins.DNB_DE.config", line 13, in <module>
ImportError: cannot import name 'Qt' from 'qt' (C:\Program Files\Calibre2\app\bin\python-lib.bypy.frozen\qt\__init__.pyc)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "calibre\ebooks\metadata\sources\identify.py", line 47, in run
File "calibre_plugins.DNB_DE.__init__", line 75, in identify
File "calibre_plugins.DNB_DE.__init__", line 58, in load_config
File "calibre\customize\zipplugin.py", line 203, in exec_module
File "calibre_plugins.DNB_DE.config", line 15, in <module>
ModuleNotFoundError: No module named 'PyQt4'

hth and thanks for the helpful plugin

-gr

[citronalco]

Thanks for the bug report. I completely missed the release of Calibre 6...
I am currently on holiday and probably won't have time to look into this until I'm back, in a few days.

In case someone has a fix in the meantime: Feel free to open a pull request https://github.com/citronalco/calibre-dnb

[citronalco]

Can someone of you test the attached (hopefully) fixed version and give feedback?

[Wiggo]

Thank you so much again, citronalco!

I tested it on some books and it works

[citronalco]

Glad to hear that.

Ok then, fixed version released.

[igorius]

Again the plugin stopped working beginning with calibre 6.17:

****************************** DNB_DE Covers ****************************** 
Request extra headers: [('User-agent', 'Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko')] 
Failed to download valid cover 
Took 108.48140907287598 seconds 
No cached cover found, running identify
Query String: num=9783641294229 AND num=9783641294229 NOT (mat=film OR mat=music OR mat=microfiches OR cod=tt)
Query URL: https://services.dnb.de/sru/dnb?version=1.1&maximumRecords=100&operation=searchRetrieve&recordSchema=MARC21-xml&query=num%3D9783641294229%20AND%20num%3D9783641294229%20NOT%20%28mat%3Dfilm%20OR%20mat%3Dmusic%20OR%20mat%3Dmicrofiches%20OR%20cod%3Dtt%29
Got records: 1
Parsing records
[776.w] Found other issue with IDN 1272076784
Query String: num=1272076784 NOT (mat=film OR mat=music OR mat=microfiches OR cod=tt)
Query URL: https://services.dnb.de/sru/dnb?version=1.1&maximumRecords=100&operation=searchRetrieve&recordSchema=MARC21-xml&query=num%3D1272076784%20NOT%20%28mat%3Dfilm%20OR%20mat%3Dmusic%20OR%20mat%3Dmicrofiches%20OR%20cod%3Dtt%29
Got records: 1
[264.b] Publisher: Ludwig Buchverlag
[264.c] Publication Year: 2023-01-01 12:30:00
[245] Title: Waldwissen : Vom Wald her die Welt verstehen. Erstaunliche Erkenntnisse über den Wald, den Menschen und unsere Zukunft - Das umfassendste Buch zum Thema Wald – Standardwerk
[245] Title_Sort: Waldwissen : Vom Wald her die Welt verstehen. Erstaunliche Erkenntnisse über den Wald, den Menschen und unsere Zukunft - Das umfassendste Buch zum Thema Wald – Standardwerk
[100.a] Primary Authors: Wohlleben, Peter
[700.a] Secondary Authors: Ibisch, Pierre ˜L.œ
[856.u] Trying to download Comments from: http://deposit.dnb.de/cgi-bin/dokserv?id=19e848cc24f747718c6a2ed32619c956&prov=M&dok_var=1&dok_ext=htm
[856.u] Got Comments: <p><em>Springe direkt zu:</em></p>
<ul>
<li><a href="DE/Benutzung/inhaltstexte.html?nn=56454#content">Inhalt</a></li>
<li><a href="DE/Benutzung/inhaltstexte.html?nn=56454#navPrimary">Hauptmenü</a></li>
<li><a href="DE/Benutzung/inhaltstexte.html?nn=56454#search">Suche</a></li>
<snipped code from website>
No cover found

and

****************************** DNB_DE (3, 2, 2) ****************************** 
Found 1 results 
Downloading from DNB_DE took 101.36612796783447 


--- 
Title               : Jeder liest Drecksack/Everyone's Reading Bastard : zweisprachige Ausgabe
Title sort          : Jeder liest Drecksack/Everyone's Reading Bastard : zweisprachige Ausgabe
Author(s)           : Nick Hornby [Hornby, Nick]
Publisher           : Köln ; eBook by Kiepenheuer&Witsch
Tags                : Erzählende Literatur
Languages           : de
Published           : 2014-01-01T11:30:00+00:00
Identifiers         : isbn:9783462307993, urn:nbn:de:101:1-2014092716662, dnb-idn:1058795074, ddc:820|B
Comments            : <p><em>Springe direkt zu:</em></p>
<ul>
<li><a href="DE/Benutzung/inhaltstexte.html?nn=56454#content">Inhalt</a></li>
<snipped code from website>

and finally with calibre 6.18:

****************************** DNB_DE (3, 2, 2) ******************************
Found 0 results
Downloading from DNB_DE took 30.34705400466919
Query String: num=9783732587971 AND num=9783732587971 NOT (mat=film OR mat=music OR mat=microfiches OR cod=tt)
Query URL: https://services.dnb.de/sru/dnb?vers...%20cod%3Dtt%29
ERROR: Got invalid response:
Plugin DNB_DE failed
Traceback (most recent call last):
File "mechanize/_urllib2_fork.py", line 1238, in do_open
File "http/client.py", line 1374, in getresponse
File "http/client.py", line 318, in begin
File "http/client.py", line 279, in _read_status
File "socket.py", line 705, in readinto
File "ssl.py", line 1273, in recv_into
File "ssl.py", line 1129, in read
TimeoutError: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "calibre_plugins.DNB_DE.__init__", line 1093, in execute_query
data = self.browser.open_novisit(queryUrl, timeout=timeout).read()
File "mechanize/_mechanize.py", line 241, in open_novisit
File "mechanize/_mechanize.py", line 287, in _mech_open
File "mechanize/_opener.py", line 193, in open
File "mechanize/_urllib2_fork.py", line 425, in _open
File "mechanize/_urllib2_fork.py", line 414, in _call_chain
File "calibre/utils/browser.py", line 28, in https_open
File "mechanize/_urllib2_fork.py", line 1240, in do_open
urllib.error.URLError: <urlopen error The read operation timed out>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "calibre_plugins.DNB_DE.__init__", line 1112, in execute_query
xmlData.find('diagnostics/diag:diagnostic/diag:details', namespaces={
AttributeError: 'NoneType' object has no attribute 'find'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "calibre/ebooks/metadata/sources/identify.py", line 47, in run
File "calibre_plugins.DNB_DE.__init__", line 117, in identify
results = self.execute_query(log, query, timeout)
File "calibre_plugins.DNB_DE.__init__", line 1121, in execute_query
log.error(data)
UnboundLocalError: local variable 'data' referenced before assignment

[citronalco]

Originally Posted by igorius

Again the plugin stopped working beginning with calibre 6.17

Currently DNB has some trouble with their servers (see the "Störung" note at https://www.dnb.de/DE/Home/home_node.html).
I hope that's the cause for the issues.

[citronalco]

DNB removed the system outage message from their webpage.
But their API still has problems: Some responses come quite fast, others take dozens of seconds, or even time out.

And "for technical reasons" they disabled access to the content descriptions.

Seems like DNB has bigger difficulties.

[citronalco]

DNB has put the "Störung" (system outage) label on it's web page again

[Wiggo]

Yep. It looks like having real problems

[citronalco]

Created an update to work around DNB's long long-lasting issues (or changes?):

1. Currently instead of a book's content, DNB returns an error page. This error page is now ignored and not stored as "comments".
2. DNB takes much longer to respond, up to several seconds. Either they're under load or have introduced some kind of delay. For sure it's no browser agent detection, I tried that.
So for now this plugin only asks DNB for max. 10 search results (previously: 100). Each of the results triggers some requests, so it should be considerable faster now.

I hope that works!

[Wiggo]

It works again - thank you so much!

[beedaddy]

The DNB plugin doesn't fetch covers here. The log always says "No cover found", but it does find some other metadata:

Starting cover download for: Babel
Query: Babel ['R.F. Kuang'] {'isbn': '9783751748384'}

****************************** DNB_DE Covers ******************************
Request extra headers: [('User-agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36')]
Failed to download valid cover
Took 2.150336742401123 seconds
No cached cover found, running identify
Query String: num=9783751748384 AND num=9783751748384 NOT (mat=film OR mat=music OR mat=microfiches OR cod=tt)
Query URL: https://services.dnb.de/sru/dnb?vers...%20cod%3Dtt%29
Got records: 1
Parsing records
[776.w] Found other issue with IDN 1275421067
Query String: num=1275421067 NOT (mat=film OR mat=music OR mat=microfiches OR cod=tt)
Query URL: https://services.dnb.de/sru/dnb?vers...%20cod%3Dtt%29
Got records: 1
[264.b] Publisher: Eichborn
[264.c] Publication Year: 2023-01-01 12:30:00
[245] Title: Babel : Roman | »Das Aufregendste im Fantasygenre seit Harry Potter« Denis Scheck | Deutsche Ausgabe
[245] Title_Sort: Babel : Roman | »Das Aufregendste im Fantasygenre seit Harry Potter« Denis Scheck | Deutsche Ausgabe
[100.a] Primary Authors: Kuang, R. F.
[856.u] Trying to download Comments from: http://deposit.dnb.de/cgi-bin/dokser...=1&dok_ext=htm
[856.u] Got Comments:
»Das Aufregendste im Fantasygenre seit Harry Potter« Denis Scheck
[...]

[016.a] Identifier IDN: 1287930875
[024.a] Identifier URN: nbn:de:101:1-2023050402454601628904
[020.a] Identifier ISBN: 9783751748384
[082.a] Indentifiers DDC: 830,B
[600.a-655.a] Non-GND Subjects: Oxford Empire Dark Academia Kolonialismus Großbritannien UK Humor britischer Humor London Literarische Unterhaltung Magie China Kanton Übersetzung Universität Verschwörung Widerstand Kolonialreich Young Adult Fantasy Neil Gaiman Phantastik Fantastik phantastische Literatur Fantasy Bücher Fantasy Bücher Fantasy
[250.a] Edition: 1. Auflage 2023
[041.a] Languages: de
[020.a ALTERNATE] Identifier ISBN: 9783847901433
Final formatted result:
Title : Babel : Roman | »Das Aufregendste im Fantasygenre seit Harry Potter« Denis Scheck | Deutsche Ausgabe
Title sort : Babel : Roman | »Das Aufregendste im Fantasygenre seit Harry Potter« Denis Scheck | Deutsche Ausgabe
Author(s) : R. F. Kuang [Kuang, R. F.]
Publisher : Köln ; Eichborn
Tags : Oxford, Empire, Dark Academia, Kolonialismus, Großbritannien, UK, Humor, britischer Humor, London, Literarische Unterhaltung, Magie, China, Kanton, Übersetzung, Universität, Verschwörung, Widerstand, Kolonialreich, Young Adult, Fantasy, Neil Gaiman, Phantastik, Fantastik, phantastische Literatur, Fantasy Bücher
Languages : de
Published : 2023-01-01T11:30:00+00:00
Identifiers : isbn:9783751748384, urn:nbn:de:101:1-2023050402454601628904, dnb-idn:1287930875, ddc:830|B
Comments :
»Das Aufregendste im Fantasygenre seit Harry Potter« Denis Scheck
[...]

-----
No cover found

************************************************** ******************************

Is this a known problem or am I the only one?

[beedaddy]

I've no clue if I did the right thing, but I managed to get it work again (for me). I did a PR to the github repo of the plugin, so let's see what happens.

[citronalco]

Originally Posted by beedaddy

I've no clue if I did the right thing, but I managed to get it work again (for me). I did a PR to the github repo of the plugin, so let's see what happens.

Thanks! I've released a fixed version.