[Metadata Source Plugin] Deutsche Nationalbibliothek

[citronalco]

This plugin uses the catalogue of the Deutsche Nationalbibliothek (DNB) to retrieve metadata.
DNB is the German central archival library. German publishers are required to send them a copy of every book for archival, so it's the largest metadata source for literature published in Germany.

Main Features
This plugin supports retrieval of all relevant metadata (authors, title, cover, ISBN, edition, tags, publication date, languages, publisher, comments, series, series index, URN and DNB-IDN).
For books without series information it can try to extract series and series index from the title. Standardized GND subjects and/or non-GND subjects can be used as tags.
This plugin works with Python 2 and Python 3.

Limitations
The returned publication date contains only the year, not the precise date.

Installation
Download the attached zip file and install the plugin as described in the Introduction to plugins thread.
The plugin is also available in Calibre's Plugin Updater.

How to report Bugs
If you find any issues please report them in this thread or on Github.

Version History

Spoiler:

v2.0.6 - 11 Jul 2018

• First version posted on this forum

v2.0.7 - 5 Oct 2018

• Add some error reporting

v2.1.0 - 4 Jan 2019

• Fix retrieval of publication date and publisher name for new Calibre versions
• Improve series extraction
• Code cleanup

v2.1.1 - 5 Jan 2019

• Further improve series extraction

v2.1.2 - 7 Oct 2019

• Fix retrieval by command line
• Tidy up query creation, at least a bit
• Workaround for SRU issue with ? and + signs
• Improve retrieval of series info from metadata
• Improve series guesser

v2.1.3 - 17 May 2020

• Fix regression when searching without author
• Improve series detection
• Preliminary support for Python 3

v3.0.0 - 9 Jul 2020

• Full support for Python 2 and Python 3

v3.0.1 - 29 March 2021

• Remove web scraping and SRU token: Beginning of April 1st 2021 DNB's SRU interface is publicly available
• If ISBN is given only search for the ISBN and ignore title and authors. This is how other plugins handle this, too
• Improve XPATH code
• Fix duplicate "urn:urn:" prefix at URN

v3.1.1 - 10 April 2021

• Improve logging
• Fix language extraction
• Reorder queries to improve results quality
• Fix filtering out audio books
• Fix multiline publisher location
• Improve creation of title variants for queries
• Improve handling of titles with numbers with leading zeros
• Don't query DNB for more results when good results are already found
• Update minimum required Calibre version to 0.9.33

v3.1.2 - 10 May 2021

• Fix regression in series extraction
• Fix skipping of audio books
• Minor improvements

v3.1.3 - 30 December 2021

• Fix cover image download (DNB has changed its URL)
• Slightly improve search

v3.1.4 - 2 January 2022

• Fix syntax error in broken v3.1.3

v3.1.5 - 13 January 2022

• Change author naming scheme from "lastname, firstname" to "firstname lastname", similar to most other plugins
• Include all author names into "Authors Sort", not only the first one

v3.1.6 - 30 January 2022

• Output metadata in normalized UTF-8 form (NFC)

v3.1.7 - 31 March 2022

• Skip TLS cert validation as CA is missing in some Python installations

v3.2.0 - 23 July 2022

• Support for Calibre 6
• Increase minimum required Calibre version to 3.48.0. Older versions have issues with TLS and can't download any data from DNB.

v3.2.1 - 22 August 2022

• Fix retrieval of non-GND subjects

v3.2.2 - 12 February 2023

• Fix bug preventing cover downloads
• Rework query creation for better results
• Get comments and cover also from other physical editions

v3.2.3 - 19 June 2023
Workaround DNB's issues

• DNB takes up to several seconds to respond, so only query max. 10 results (prev: 100)
• Instead of a book's content, DNB returns an error page. Ignore this page.

v3.2.4 - 01 February 2024

• Fix downloading covers for books having no dnb-idn saved

[Lawer]

Hello! This plugin sounds so good.

Register at Deutsche Nationalbibliothek was so easy.

But i´m not sure, what kind of sru access token is necessary.

What kind of format ? Marc 21 or RDF ? What meens that?
What kind of archive I need ? With or without GND???

I´m really helpless!

[Lawer]

Finally i´ve got the sru Access Token!

I wrote the sru Access token Code in the Plugin - but it didn´t work.

No Metadatas found.

Without the sru Access token Code it works!

What´s wrong?

[citronalco]

First of all, Lawer's issue with the non working SRU token was a non-issue, turned out to be a copy-paste problem.

The description of the steps required to get a SRU Access Token was a little too vague:
When requesting the token tell them you want to have access to the title data catalogue ("Titeldaten-Katalog") in the format MARC21-XML. I have updated the README file accordingly.

Marc21, Marc21-XML and RDF are different data formats, just like JPG and PNG.
DNB offers several catalogues, the Titeldaten-Katalog contains book information (title, author, publication date,...), the GND ("Gemeinsame Normdatei") contains e.g. biographic information about authors, valid descriptive tags etc.

[citronalco]

In the meantime (2018) I've updated the plugin a few times and fixed a few bugs some helpful guys reported to me.
It's now also available in a Python 3 variant: https://github.com/citronalco/calibre-dnb/releases

Maybe I should note that the DNB is the German central archival library. Every publisher is required to hand them a book over for archival. So it's the largest metadata source for German literature.

[collaco]

Just installed it - got my access token from DNB in less then 48h - works perfect! Thanx

[citronalco]

The plugin now supports Python 2 and Python 3. No different ZIP files required.

Fixes:
- Fixed downloading metadata in Python 3 without SRU.

You can download the newest release from here:
https://github.com/citronalco/calibre-dnb/releases

If you find any issues please report them on Github.

[citronalco]

Does anyone know how I get my plugin listed on the plugins page (https://plugins.calibre-ebook.com/)?

[BetterRed]

Read the Notes for developers at the top of the Index of plugins thread.

You'll need to edit the opening post of this thread so that conforms to the layout and content of other plugin threads - use one of the other Metadata Download plugin's first post as a guide - main thing is that the plugin zip must be attached to the first post.

The next release of calibre will be released soon. It requires adjustments to be made to plugin code to make them python 3 compatible, see the relevant sticky threads at the top of this forum for details.

When you're ready to roll, PM me or one of the other mods with the details you want in the index.

BR

[BetterRed]

@citronalco - you should have edit permission now.

Paul has attached the plugin to the first post for you, but you probably want to add a couple of items, like Version History etc. And maybe update the plugin to be Python 3 compliant.

Could you remove or move the link to Gitub in the first post, it could confuse users who are accustomed to getting their plugins from here or via calibre itself.

I see you've done the P3 changes - I missed that.

Send me a PM with the details to put in the index when you're ready..

BR

[beedaddy]

@citronalco I certainly did something wrong and I hope you can give me a hint: The DNB gave me a access token, but fetching metadata only returns me a cover image but no other metadata like comments or so. What could be the reason for that? (And I wonder what I should select for "fetch subjects"…)

[citronalco]

Originally Posted by beedaddy

@citronalco I certainly did something wrong and I hope you can give me a hint: The DNB gave me a access token, but fetching metadata only returns me a cover image but no other metadata like comments or so. What could be the reason for that? (And I wonder what I should select for "fetch subjects"…)

There are several things you could check:
- In the generic metadata configuration dialogue (the window where you can select which metadata plugin you want to use): Have you selected any metadata fields to download?
- The plugin returns the same metadata, albeit much slower, without a token (token field empty in the config window). Do you get metadata without a token? If yes: Check if you've entered the token correctly (maybe you accidentally pasted a space at the end or similar)
- When retrieving metadata has finished you could have a look at the protocol and search for the "Query URL". You can simply paste the URL into a browser and check if you get a longish XML file or a error message.

If nothing of the above solves your problem: Please send me the protocol of a search.

Just in this moment DNB seems to have issues with their API anyway: Currently they don't check the token at all and accept anything - Maybe they simply have issues with their infrastructure. I doubt they intentionally opened their API for unauthenticated users - though I'd be happy if they would.

About the subjects:
There are two types of subjects:
- GND subjects are selected by librarians from a standardized catalogue (the "Gemeinsame NormDatei"), so they are are not available for every book, but they are usually high quality
- Non GND subjects are simply delivered by the publisher and can be anything. There's no quality checking done by DNB, so it can be something useful or just marketing SEO crap.
Older books usually do not have non-GND but GND subjects, with newer books it's the other way round.
Personally I prefer the setting "GND subjects if available, otherwise non-GND subjects", but it's a matter of taste.

[Henning]

@citronalco

For your information:
Email from "Schnittstelle-Service:
BITTE BEACHTEN SIE: Zukünftig wird die Nutzung der SRU-Schnittstelle ohne vorherige Registrierung möglich sein. Die technische Umstellung ist Anfang März bereits erfolgt, so dass Sie jetzt schon auf die gewünschte Schnittstelle zugreifen können. Die vollständige Umstellung wird am 1. April vollzogen sein und dann auch auf der Webseite entsprechend kommuniziert, Sie können die Schnittstelle aber ab sofort nutzen und müssen diesbezüglich nichts mehr tun.

Wir informieren regelmäßig über wichtige Änderungen und Neuerungen an den Schnittstellen der Deutschen Nationalbibliothek (DNB). Ebenso halten wir Sie zeitnah über Störungen und Wartungsarbeiten auf dem Laufenden.
Wenn Sie diesen Service nutzen möchten, melden Sie sich bitte für den Mailverteiler an.*
SRU-Schnittstelle: https://lists.dnb.de/mailman/listinfo/sru-schnittstelle


==> as far as I understand: the token-system is outdated - Your wish to establish a system without registration (and token) has come thru.

[beedaddy]

Thanks @citronalco for your detailed explanation. I will double-check the settings. I think that sometimes DNB simply lacks the blurb.

[citronalco]

@Henning: Good news, thanks!