Download options for Identifier (isbn, asin, etc.) metadata

[digital_carver]

It seems like every metadata provider has their own identifier number (asin, google, goodreads, etc.), and they send that in along with the other pieces of metadata. Currently, Calibre doesn't allow not downloading this piece of metadata (apparently for some internal technical reason?), which means that in many cases the existing metadata is overwritten by some custom piece of identifier.

For example, I'm right now in the middle of going through my bulk metadata download results, and in 90% of the cases that require change, the change is that the local "mobi-asin" is about to be overwritten by "google" or "goodreads" identifiers. I wish to use the tags and such metadata from these providers, so disabling them entirely is not a solution. So in every case I tab to the existing identifier field, Ctrl-c, shift-tab twice to get back to downloaded identifier, and paste it there (I wish to merge, not overwrite, so the arrow button doesn't help.)

If it's not possible to disable the identifier metadata download, could an option be added to merge the existing identifier with the new ones, if the new ones don't contain an instance of the same identifier? (ie., merge existing local "mobi-asin:yyy" if downloaded data is "goodreads:xxx", but overwrite local "mobi-asin:yyy" if downloaded data is "goodreads:xxx,mobi-asin:zzz").

Is something like this already possible through some preference or plugin? If not, please consider adding it as an option.

[theducks]

Which is why I usually use Ctrl-D instead of the one built into Metadata editor. You see before (on right) and proposed (on Left) with an arrow to overwrite proposed with the old (Except you cant force it blank).
Blank proposed, always gets the old

Remember to configure EACH metadata source

[digital_carver]

> You see before (on right) and proposed (on Left) with an arrow to overwrite proposed with the old

Yep, that's the screen I'm in. The issue is that I don't want to overwrite the identifier in most cases, rather wish to merge them: for eg., when the download has asin and google identifiers, my local file has isbn and mobi-asin already, the best of both worlds would be to merge them. But currently the only way to do that is to manualy copy and paste and add commas and all that, and do this again and again for every book.

I guess the original idea was that identifier would be a single piece of data, but now because of several sources we have, it's actually a combination of multiple pieces of metadata. So it doesn't always make sense to "update" or "correct" the local existing identifier with the downloaded one. That's the reasoning behind my suggestion above: treat the identifiers individually rather than as a single piece of string during metadata download, so that for eg. local "asin" would be overwritten by remote "asin", but local "isbn" wouldn't be affected by remote "asin" download, instead just getting merged with it.

[BetterRed]

@digital_carver - FWIW - the Ctrl/D form of Metadata download (which can be used on a single book, as well as bulk) has a long press option on the IDs column [←] button to Merge the values.

BR

[digital_carver]

@BetterRed While looking at why my Calibre didn't have that long-press affordance at the place you mention, I found that I was running an ancient version of Calibre - 1.48, which seems to be the last .deb-packaged version of Calibre on any Linux repositories! I had assumed my regular 'apt-get upgrade's would have taken care of keeping Calibre updated, never knew that method had been abandoned. I'm now downloading the latest version through the recommended Linux install method.

And what you mentioned certainly a nice-to-have feature, but probably takes the same amount of time as manually copy-pasting it using keyboard shortcuts takes for me. But I'll give it a whirl once I get the new Calibre set-up with my preferences. (Don't want to just copy the config directory between major version changes, since that's already unsupported and might mess things up.)

[BetterRed]

Originally Posted by digital_carver

@BetterRed While looking at why my Calibre didn't have that long-press affordance at the place you mention, I found that I was running an ancient version of Calibre - 1.48, which seems to be the last .deb-packaged version of Calibre on any Linux repositories! I had assumed my regular 'apt-get upgrade's would have taken care of keeping Calibre updated, never knew that method had been abandoned. I'm now downloading the latest version through the recommended Linux install method.

And what you mentioned certainly a nice-to-have feature, but probably takes the same amount of time as manually copy-pasting it using keyboard shortcuts takes for me. But I'll give it a whirl once I get the new Calibre set-up with my preferences. (Don't want to just copy the config directory between major version changes, since that's already unsupported and might mess things up.)

I would try using the existing config data, if you run into problems you can always reset it by running the Welcome Wizard - it's at the bottom right of Preferences.

I'm likewise not fond of mouse clicking - long or short. You can focus on the Identifiers button with Alt+I, Tab, Tab but then what - I don't think there's a keyboard equivalent of long press, in Windows at least.

BR

[digital_carver]

Originally Posted by BetterRed

I'm likewise not fond of mouse clicking - long or short. You can focus on the Identifiers button with Alt+I, Tab, Tab

Yup, that's basically what I did: Alt-I, tab, tab, Ctrl-C, shift-Tab, shift-Tab, End, comma, Ctrl-V, Alt-N

Went through my library of ~700 books this way, sometimes doing this, sometimes Alt-R, Alt-R, Enter (to reject), sometimes just Alt-I, delete (when my local data was a superset of the downloaded stuff). With some instrumental music running in the background, it was practically a Zen state.

Having to do this manually also helped me reject some pretty significant mistakes in the downloaded metadata - data of wrong book with somewhat similar title, sometimes even wrong author - so I guess even with such a merging feature I can't just let it do things automatically. (Though I wonder if my older version was to blame - I did these a while before discovering I was a few years out of date, so perhaps there's better algorithms in current versions.)

[eschwartz]

Funny, I always thought the identifiers would be merged by themselves. It does seem the logical way to do things.

And doing a metadata download right now, and scrutinizing the value of the metadata before and after, shows me I am right -- the mobi-asin is preserved.


mobi-asin will never be downloaded -- it is calibre-specific, used to store the value of the exth 113 MOBI header (a/k/a ASIN). So if you have ever downloaded metadata for a book, and that identifier is still there, it must've been preserved... and I have many such books.


I cannot promise what outdated and downstream-patched versions of calibre will do (yes, calibre is patched downstream by many distros, which sometimes breaks functionality. e.g. cherrypy and content servers with paswords.)

[kovidgoyal]

Long click the arrow button and you will get a popup menu with an option to merge the identifiers.

[digital_carver]

Originally Posted by eschwartz

Funny, I always thought the identifiers would be merged by themselves. It does seem the logical way to do things.

And doing a metadata download right now, and scrutinizing the value of the metadata before and after, shows me I am right -- the mobi-asin is preserved.

I too tried it out now, and it does seem to already do what I suggested: overwrite similar identifiers, merge dissimilar ones. Neat. (I don't know if this was true of the ancient version I was using during the previous download, but it works in my current 2.37.1 Calibre.)

In fact, merging seems to happen with Tags too, which (afaik) seems undocumented. The existing tags remain, and the new tags get added additionally. For eg., with "The Autobiography of Benjamin Franklin", this was the tags metadata before download:

Franklin, Statesmen -- United States -- Biography, Politics

This was the tags shown in the left side during "Review downloaded metadata":

History, Biography, Classics, Politics, Non-Fiction

And this is the final tags list after pressing Ok on that dialog:

Classics, non-fiction, Biography, Politics, Franklin, Statesmen -- United States -- Biography, History

As you can see, it's a merge rather than an overwrite.

The Configure Metadata Download dialog has only an "Append comments" option related to this, nothing about tags or identifiers, so it's not really obvious what all is going on in what field during the update.

[eschwartz]

Identifiers and tags in calibre manifest as lists.
Comments are a single string of HTML.


Rule of -- lists get merged.

[BetterRed]

Originally Posted by eschwartz

Identifiers and tags in calibre manifest as lists.
Comments are a single string of HTML.

Rule of -- lists get merged.

- state of mind

Bulk download Tags and Identifier columns:

When I do a left mouse button click on the blue ← button (or hit space if it has focus) the existing values replace the downloaded values, which is what the button tooltip implies it will do - Discard the downloaded value for : <column name>

The same text appears as the first item in the dropdown list I get when I do a long left mouse button press on blue ← button.

To get values to merge I must do a long left mouse button press on blue ← button and select the second item in the dropdown list - Merge <column name>

Aside : I'd rather have two buttons on the list columns - blue ← button (Discard downloaded values) and aqua jaws button (Merge values). IMO it would be less confusing and consistent with the other blue ← buttons; they would also be keyboard-able . . .

BR

[eschwartz]

@BR,

I am aware there is a right-click option to merge, but at least for me (and @ digital_carver it would seem), simply leaving it alone ALSO merges them.

[BetterRed]

Originally Posted by eschwartz

@BR,

I am aware there is a right-click option to merge, but at least for me (and @ digital_carver it would seem), simply leaving it alone ALSO merges them.

I hadn't tried right button click, I read somewhere it needed a long left button mouse press, being mouse (and touch) averse that is not something I would think of doing 'naturally'. But I did try the windows menu key which always does what a right click does everywhere else in and out of calibre - except on these two buttons

Nevertheless for me at least a click on the Tags or Identifier blue ← does what the tooltip says it will do - discard the downloaded values - what do you see in the tooltip ?

It guess it could be a Windows 10 aberration - but I would be gobsmacked if it is.

BR

[eschwartz]

I don't click on it. I leave the right side and the left side intact -- I click Review, eyeball the results, and go straight on to Accept, do not pass Go, do not collect $200 for clicking things.

Clicking once on the arrow will reject the downloaded values and present the "result" for approval.
Clicking twice merges values and presents the results for approval.
Clicking zero times merges values after you click Accept.