Metadata download questions

CarstenF · 08-30-2015, 04:41 PM

Dear calibre group,

I've read the manual, but still please allow some questions about downloading metadata for a single book:

1. When I use Goodreads as the single metadata provider, I get a single result that comes with a great comment (and other fields). When I additionally use Amazon as the second metadata provider, I get two results, but none of them is exactly like the one obtained with Goodreads alone. Instead, it looks as if in one result, the comment is somehow composed of the individual comments from Goodread and Amazon. Is that possible, and the intended result? If so, how does this mechanism (generally) work?

2. Whenever I download metadata, all downloaded fields (whose checkbox is set so that they are accounted for) will, when OK is clicked, unconditionally overwrite/replace the previous field values. Is that correct?

3. If the answer to 2. is "yes", doesn't that imply a conflict/problem if I wanted to download e.g. the tags (for newly added books) and to manually edit them at a later time? After all, an accidental re-download would overwrite my hand-edited tags!?

(EDIT: Same problem for the rating column: When I add a new book, it's ok to download the rating along with the other metadata. After I've read a book, I would rather record my personal rating, foregoing the provider's value. Thus, the existing default column seems fine for this, but if I accidentally re-download the metadata, all is lost?)

4. Assuming that the answer to 2. is "yes", if I wanted to add my own, custom comment in addition to the default comment (e.g. a review), a much better approach than adding my comment to the existing default field is to add another, custom, dedicated column, right?

Any help would much be appreciated!

Best regards,
Carsten

eschwartz · 08-30-2015, 06:59 PM

When downloading metadata, there is a second button for configuring downloaded metadata (the gear icon right next to the download button in the Edit Metadata view).

You can choose which fields you want to download. Additionally there is an option to "append comments to existing".
But by and large, metadata download is meant to overwrite the current (assumed bad

) metadata.

When using the bulk metadata download (shortcut key is CTRL+D) you can review the metadata, which will be from the better match.
You can manually edit the new metadata (on the left) or press the <- arrow to revert a specific field to the old metadata (on the right).

I usually use the CTRL+D route, but I *thought* in Edit Metadata ==> Download you were supposed to get a choice between metadata from either source, not any kind of aggregate. I could be wrong though, as I said I don't use it enough to say with confidence what should happen.

theducks · 08-30-2015, 07:35 PM

Quote:

Originally Posted by eschwartz

When downloading metadata, there is a second button for configuring downloaded metadata (the gear icon right next to the download button in the Edit Metadata view).

You can choose which fields you want to download. Additionally there is an option to "append comments to existing".
But by and large, metadata download is meant to overwrite the current (assumed bad

) metadata.

When using the bulk metadata download (shortcut key is CTRL+D) you can review the metadata, which will be from the better match.
You can manually edit the new metadata (on the left) or press the <- arrow to revert a specific field to the old metadata (on the right).

I usually use the CTRL+D route, but I *thought* in Edit Metadata ==> Download you were supposed to get a choice between metadata from either source, not any kind of aggregate. I could be wrong though, as I said I don't use it enough to say with confidence what should happen.

I think you are correct.
In the metdata edit, you pick from the answers.
But the Overwrite / append setting still applies

BetterRed · 08-30-2015, 07:51 PM

You can also configure what items get downloaded from which sources, I think

So if you prefer Goodread's Comments but Amazon's Ratings - you would use the Configure selected source button in Preferences->Metadata download to make those selections.

BR

CarstenF · 08-31-2015, 07:22 PM

Many thanks to you all, this helped a lot!

Quote:

Originally Posted by eschwartz

You can choose which fields you want to download. Additionally there is an option to "append comments to existing".
But by and large, metadata download is meant to overwrite the current (assumed bad

) metadata.

Ok, many thanks for confirming this!

Quote:

When using the bulk metadata download (shortcut key is CTRL+D) you can review the metadata, which will be from the better match.
You can manually edit the new metadata (on the left) or press the <- arrow to revert a specific field to the old metadata (on the right).

This is a good tip! I had not used bulk metadata download before, because I wanted to understand the single "Edit Metadata" download first and assumed that the bulk would be more complicated. But alas, it turns out that the bulk download is in fact easier!

Quote:

I usually use the CTRL+D route, but I *thought* in Edit Metadata ==> Download you were supposed to get a choice between metadata from either source, not any kind of aggregate. I could be wrong though, as I said I don't use it enough to say with confidence what should happen.

I experimented a bit more (with the "Edit Metadata" single downloads), and found that the protocols provide helpful insights: If I use both Goodreads and Amazon as metadata providers for e.g. Dicken's "David Copperfield", then calibre loads

three results from Amazon
one result from Goodreads

and the final lines of the protocol say:

Code:

Merging results from different sources and finding earliest  publication dates from the worldcat.org service 
We have 2 merged results, merging took: 0.62 seconds

And in fact, the subsequent list dialog from which I can choose among the downloaded results only offers two choices, not four, both (mostly) from Amazon, where one result however also mentions a reference to Goodread. The tags seem to have been combined as well.

I think that this is quite surprising and confusing, especially because the single Goodreads result is never mentioned individually. I also understand that I can choose which fields to download, both "globally" and per provider, but not knowing the rules with which the final result is composed leaves me quite unsure. (For example: Does "metadata from the best match" enclose all fields from that match? Or will individual fields from lesser matches still be used, e.g. when the field was excluded from the best match?)

In summary, it seems that with the combination of the bulk download with its table-view and the selection of individual fields per metadata provider can solve my problems/questions.

But if I may make a suggestion: I think that it would be wonderful if metadata download results were presented in a spreadsheet-like table: one column for the old/existing data, and one column for each provider; one row for each field. Then have the user choose for each field (row) which provider's value he/she wishes to overtake (possibly with multi-selection for concatenation). Just a thought.

Again, many thanks to everyone for your help!

Best regards,
Carsten

eschwartz · 08-31-2015, 07:33 PM

Now that I think of it, I do vaguely remember something about merged results, I guess this is it. Now you know why I prefer bulk mode.

Maybe

if calibre is reasonably sure two results are about the same edition, it aggregates the metadata.

CarstenF · 09-01-2015, 04:31 PM

Can this merging/aggregating feature be turned off?

After all of the above, I tried to be clever, using this setup:

From Amazon and Google, remove all check marks in order to load nothing but the title pages;
from Goodreads load everything (all check marks checked);
but globally don't load the rating (I want to assign my own).

The intention was to get the metadata only from Goodreads, some title pages for choice from several providers, and the rating from no one.

Alas, it doesn't work, neither in the Edit Metadata (single book) nor in the bulk load:

In the Edit Metadata, it still presents 12 results -- most of them useless, as they only come with title and authors (due to my check marks settings), and the single useful result is still a merge, e.g. authors from Google or Amazon, and the comment from Goodreads.

With the bulk load, apparently it is not Goodreads that is found to be the best match, but one of the others. As I've removed so many check marks, most fields are empty and the result is useless.

What now?

I guess I can just configure Goodreads as my single and only metadata provider; I can certainly live with that. But it would be a lot more useful if the merging could just be turned off -- is that possible?

kovidgoyal · 09-01-2015, 11:14 PM

No, it is not possible.

CarstenF · 09-02-2015, 03:43 PM

Dear Kovid,

many thanks for your reply!
I guess that I'll just go then with Goodreads as my single metadata provider.

I really admire both the amount of ideas and the amount of work effort that you've put into calibre. calibre is really overwhelming both in features and flexibility, but also in options, setting and dialogs.

Being still a relatively new user, please allow me to reiterate on an outline / vision / suggestion how I imagine that metadata downloads from multiple sources could much be simplified, both for single and many books:

Metadata from multiple sources could be presented in a spreadsheet-like table. That table would quasi be a generalization of the existing "Review downloaded metadata" dialog that we can see after a bulk download:

Each field would correspond to one row,
the first column would show all locally existing values,
each downloaded metadata result would correspond to one column each,
- either each result of each provider had its own column (probably too many), or
- multiple results per provider could get merged into a single result (similar as today, but merges would consider results only of the same, common provider).

Then, for each row (field), the user could select/click/highlight the column whose value he/she wishes to overtake:

the default selection is "all cells of the first column" (the existing values, meaning: "no change"),
once done, the user's selection is saved for re-use with the next book -- the user will very likely use the same or similar settings again, so the last selection becomes the new default selection,
multiple selections per row would be possible, meaning to "merge" or "concatenate" the selected values for this field,
all this works for all kinds of field, including title page (although multi-selection is not possible for fields such as title page, rating, etc.).

Advantages:

This is very simple: For the user, it is really intuitive and easy to understand what's going on.
Full flexibility is preserved as is now.
No need to discriminate between single book and bulk metadata downloads -- they are the same (needs prev/next pages as in the single book "Edit Metadata" dialog).
It would no longer be necessary to have checkmarks with each provider that tell which fields should be loaded -- just load them all, always. The user can still ignore the related spreadsheet cells.

Disadvantages:

Needs relatively much screen space, both in width and height. Like a spreadsheet, this might require horz./vert. scrolling.

What do you think?

kovidgoyal · 09-02-2015, 10:56 PM

I disagree. Metadata is merged for identical results for a reason -- to make the process as simple as possible. Presenting the user with an explosion of results just means lots more work for every download. While it can be useful when the results are not of good quality, the reverse is true when they are.

CarstenF · 09-03-2015, 03:50 PM

Well, ok

But... the spreadsheet interface would still be nice, with the fully merged results as you say, however only one dialog for both single and bulk.

kovidgoyal · 09-03-2015, 11:19 PM

You are aware that after a single metadata download you can undo the changes to any field fo the edit metadata dialog to go back to the original value, right? Simply right click in the field and choose undo.

Terisa de morgan · 09-03-2015, 11:51 PM

Quote:

Originally Posted by kovidgoyal

You are aware that after a single metadata download you can undo the changes to any field fo the edit metadata dialog to go back to the original value, right? Simply right click in the field and choose undo.

I wasn't, only the general undo. Thank you very much

CarstenF · 09-04-2015, 06:11 PM

I wasn't aware of it, thanks for mentioning it!

However, I readily admit that I usually cannot remember the previous exact values of all fields after having downloaded new metadata, whereas seeing them next to each other is a great help: very intuitive, nothing to remember, very exact.
Well, I guess I'd still like to ask you to consider a spreadsheet-like interface as proposed above as an option for the future.

08-30-2015, 04:41 PM	#1
CarstenF Member Posts: 10 Karma: 10 Join Date: Aug 2015 Location: Germany Device: Kindle Paperwhite	Metadata download questions Dear calibre group, I've read the manual, but still please allow some questions about downloading metadata for a single book: 1. When I use Goodreads as the single metadata provider, I get a single result that comes with a great comment (and other fields). When I additionally use Amazon as the second metadata provider, I get two results, but none of them is exactly like the one obtained with Goodreads alone. Instead, it looks as if in one result, the comment is somehow composed of the individual comments from Goodread and Amazon. Is that possible, and the intended result? If so, how does this mechanism (generally) work? 2. Whenever I download metadata, all downloaded fields (whose checkbox is set so that they are accounted for) will, when OK is clicked, unconditionally overwrite/replace the previous field values. Is that correct? 3. If the answer to 2. is "yes", doesn't that imply a conflict/problem if I wanted to download e.g. the tags (for newly added books) and to manually edit them at a later time? After all, an accidental re-download would overwrite my hand-edited tags!? (EDIT: Same problem for the rating column: When I add a new book, it's ok to download the rating along with the other metadata. After I've read a book, I would rather record my personal rating, foregoing the provider's value. Thus, the existing default column seems fine for this, but if I accidentally re-download the metadata, all is lost?) 4. Assuming that the answer to 2. is "yes", if I wanted to add my own, custom comment in addition to the default comment (e.g. a review), a much better approach than adding my comment to the existing default field is to add another, custom, dedicated column, right? Any help would much be appreciated! Best regards, Carsten Last edited by CarstenF; 08-30-2015 at 04:50 PM.

09-01-2015, 04:31 PM	#7
CarstenF Member Posts: 10 Karma: 10 Join Date: Aug 2015 Location: Germany Device: Kindle Paperwhite	Can this merging/aggregating feature be turned off? After all of the above, I tried to be clever, using this setup: From Amazon and Google, remove all check marks in order to load nothing but the title pages; from Goodreads load everything (all check marks checked); but globally don't load the rating (I want to assign my own). The intention was to get the metadata only from Goodreads, some title pages for choice from several providers, and the rating from no one. Alas, it doesn't work, neither in the Edit Metadata (single book) nor in the bulk load: In the Edit Metadata, it still presents 12 results -- most of them useless, as they only come with title and authors (due to my check marks settings), and the single useful result is still a merge, e.g. authors from Google or Amazon, and the comment from Goodreads. With the bulk load, apparently it is not Goodreads that is found to be the best match, but one of the others. As I've removed so many check marks, most fields are empty and the result is useless. What now? I guess I can just configure Goodreads as my single and only metadata provider; I can certainly live with that. But it would be a lot more useful if the merging could just be turned off -- is that possible?

09-02-2015, 03:43 PM	#9
CarstenF Member Posts: 10 Karma: 10 Join Date: Aug 2015 Location: Germany Device: Kindle Paperwhite	Dear Kovid, many thanks for your reply! I guess that I'll just go then with Goodreads as my single metadata provider. I really admire both the amount of ideas and the amount of work effort that you've put into calibre. calibre is really overwhelming both in features and flexibility, but also in options, setting and dialogs. Being still a relatively new user, please allow me to reiterate on an outline / vision / suggestion how I imagine that metadata downloads from multiple sources could much be simplified, both for single and many books: Metadata from multiple sources could be presented in a spreadsheet-like table. That table would quasi be a generalization of the existing "Review downloaded metadata" dialog that we can see after a bulk download: Each field would correspond to one row, the first column would show all locally existing values, each downloaded metadata result would correspond to one column each, either each result of each provider had its own column (probably too many), or multiple results per provider could get merged into a single result (similar as today, but merges would consider results only of the same, common provider). Then, for each row (field), the user could select/click/highlight the column whose value he/she wishes to overtake: the default selection is "all cells of the first column" (the existing values, meaning: "no change"), once done, the user's selection is saved for re-use with the next book -- the user will very likely use the same or similar settings again, so the last selection becomes the new default selection, multiple selections per row would be possible, meaning to "merge" or "concatenate" the selected values for this field, all this works for all kinds of field, including title page (although multi-selection is not possible for fields such as title page, rating, etc.). Advantages: This is very simple: For the user, it is really intuitive and easy to understand what's going on. Full flexibility is preserved as is now. No need to discriminate between single book and bulk metadata downloads -- they are the same (needs prev/next pages as in the single book "Edit Metadata" dialog). It would no longer be necessary to have checkmarks with each provider that tell which fields should be loaded -- just load them all, always. The user can still ignore the related spreadsheet cells. Disadvantages: Needs relatively much screen space, both in width and height. Like a spreadsheet, this might require horz./vert. scrolling. What do you think?

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Regarding using metadata objects in identify method of metadata download plugin api	aprekates	Development	1	07-06-2014 03:35 AM
metadata download plugin related questions	cerda	Development	3	07-25-2013 05:16 PM
Failed to download metadata (both metadata & cover)	EddieSean	Calibre	0	01-31-2013 09:49 PM
[Metadata Download Plugin] Goodreads Metadata Deprecated	kiwidude	Plugins	30	04-23-2011 02:10 PM
Does "Download Metadata & Covers" also download social metadata?	iridius	Library Management	3	02-22-2011 12:50 PM

08-30-2015, 06:59 PM	#2
eschwartz Ex-Helpdesk Junkie Posts: 19,421 Karma: 85400180 Join Date: Nov 2012 Location: The Beaten Path, USA, Roundworld, This Side of Infinity Device: Kindle Touch fw5.3.7 (Wifi only)	When downloading metadata, there is a second button for configuring downloaded metadata (the gear icon right next to the download button in the Edit Metadata view). You can choose which fields you want to download. Additionally there is an option to "append comments to existing". But by and large, metadata download is meant to overwrite the current (assumed bad ) metadata. When using the bulk metadata download (shortcut key is CTRL+D) you can review the metadata, which will be from the better match. You can manually edit the new metadata (on the left) or press the <- arrow to revert a specific field to the old metadata (on the right). I usually use the CTRL+D route, but I thought in Edit Metadata ==> Download you were supposed to get a choice between metadata from either source, not any kind of aggregate. I could be wrong though, as I said I don't use it enough to say with confidence what should happen.

08-30-2015, 07:51 PM	#4
BetterRed null operator (he/him) Posts: 21,730 Karma: 29711016 Join Date: Mar 2012 Location: Sydney Australia Device: none	You can also configure what items get downloaded from which sources, I think So if you prefer Goodread's Comments but Amazon's Ratings - you would use the Configure selected source button in Preferences->Metadata download to make those selections. BR

08-31-2015, 07:33 PM	#6
eschwartz Ex-Helpdesk Junkie Posts: 19,421 Karma: 85400180 Join Date: Nov 2012 Location: The Beaten Path, USA, Roundworld, This Side of Infinity Device: Kindle Touch fw5.3.7 (Wifi only)	Now that I think of it, I do vaguely remember something about merged results, I guess this is it. Now you know why I prefer bulk mode. Maybe if calibre is reasonably sure two results are about the same edition, it aggregates the metadata.

09-01-2015, 11:14 PM	#8
kovidgoyal creator of calibre Posts: 45,356 Karma: 27182818 Join Date: Oct 2006 Location: Mumbai, India Device: Various	No, it is not possible.

09-02-2015, 10:56 PM	#10
kovidgoyal creator of calibre Posts: 45,356 Karma: 27182818 Join Date: Oct 2006 Location: Mumbai, India Device: Various	I disagree. Metadata is merged for identical results for a reason -- to make the process as simple as possible. Presenting the user with an explosion of results just means lots more work for every download. While it can be useful when the results are not of good quality, the reverse is true when they are.

09-03-2015, 03:50 PM	#11
CarstenF Member Posts: 10 Karma: 10 Join Date: Aug 2015 Location: Germany Device: Kindle Paperwhite	Well, ok But... the spreadsheet interface would still be nice, with the fully merged results as you say, however only one dialog for both single and bulk.

09-03-2015, 11:19 PM	#12
kovidgoyal creator of calibre Posts: 45,356 Karma: 27182818 Join Date: Oct 2006 Location: Mumbai, India Device: Various	You are aware that after a single metadata download you can undo the changes to any field fo the edit metadata dialog to go back to the original value, right? Simply right click in the field and choose undo.

09-04-2015, 06:11 PM	#14
CarstenF Member Posts: 10 Karma: 10 Join Date: Aug 2015 Location: Germany Device: Kindle Paperwhite	I wasn't aware of it, thanks for mentioning it! However, I readily admit that I usually cannot remember the previous exact values of all fields after having downloaded new metadata, whereas seeing them next to each other is a great help: very intuitive, nothing to remember, very exact. Well, I guess I'd still like to ask you to consider a spreadsheet-like interface as proposed above as an option for the future.