What exactly does merge do?

[bigpallooka]

I'm trying to decide which method to use when adding multiple books to a library that contains duplicates and also how to deal with duplicates. There are many posts and excellent advice on all sorts of issues to deal with duplicates. Unfortunately, I can't seem to find specific information on what happens during a merge. I understand this is 2 separate issues:
1) What happens during an auto-merge when adding multiple books?

and

2) What happens during a merge initiated by the user when dealing with duplicates?

Specifically:

What happens to fields in the metadata that have differing information contained such as different ISBN or covers or Tags or Comments, etc.?

Is there any way to guide the process outside of what the program is set to do? Would this be a process of the mysterious plugboards?

I am really most interested in knowing what occurs during the process so I can decide if I should manage merging manually and if it will reduce the workload when the source books metadata is of an unknown quality.

When you manually merge books of the same extension (file type) one disappears. Which one? Is it removed (deleted) from the library?

Regards

[ldolse]

I don't know about auto-merge - but for manual merges initiated by the user the Metadata is not merged - only the book files are. You choose the metadata you want to keep based on the order in which you highlight/select the duplicates.

[bigpallooka]

I thought it was clear in my examples (What happens to fields in the metadata that have differing information contained such as different ISBN or covers or Tags or Comments, etc.?) that I was talking about metadata. I never mentioned text or content but if I was not as obvious as I thought I am asking specifically about the metadata.

In my last question regarding duplicate books of the same file type/extension I will admit I was motivated by the prospect of viewing these books first to ascertain which has the best formatting. The act of merging these two books would then enable me to remove one (or multiple copies) while choosing both the file with the best metadata and the best formatting.

Thanks

[ldolse]

Sorry if I wasn't clear in my answer then - what I was trying to say is the metadata itself is not merged - one book's metadata wins over the other book as far as I can tell. The winner is based on the order in which you select the books before the merge.

I don't have any idea about two books with the same format - my guess would be the winner is also purely chosen based on the order which you selected the books.

[bigpallooka]

So it is really just a merging of book records, book X & Y into book Z with the metadata of book Z being used for all 3 reducing the total number of books by 2? Or combining multiple book records under the metadata and entry number of a specifically chosen book?

I.E. Not a merge of books but a combining under the metadata of the chosen book?

When this occurs in the auto-merge option when adding multiple books are the newly added books combined into the original version contained in the library?

[Manichean]

Try selecting multiple books and clicking the two merge options. You should, if you haven't previously disabled it, get a dialog box explaining what will be done and asking you if you want to proceed. I found the explanatory text to be pretty helpful.

[bigpallooka]

It contradicts what idolse has said and that it does in fact merge the metadata but once again, what does that actually mean? Does it somehow chose which metadata is more relevant? does it only import metadata into fields that are blank?

It does clearly state that it does not merge the ISBN and that duplicate files will be deleted permanently.

[Manichean]

From what I understand, the first selected book is the "master copy" and the metadata gets used from that book. You could ask Starson about the details, as far as I remember, he wrote the merge code.

[itimpi]

In my experience there is a limited amount of metadata merging. As an example fields that are empty can be added to the master record. Also the tags end up as a superset. Book formats are only merged if that format does not already exist in the master.

The easiest things to do is to make copies of a few of your book records and use these to do some experimentation.

[kiwidude]

Originally Posted by bigpallooka

So it is really just a merging of book records, book X & Y into book Z with the metadata of book Z being used for all 3 reducing the total number of books by 2? Or combining multiple book records under the metadata and entry number of a specifically chosen book?

I.E. Not a merge of books but a combining under the metadata of the chosen book?

When this occurs in the auto-merge option when adding multiple books are the newly added books combined into the original version contained in the library?

If you want to see exactly what it does, the code for the manual merges at least is in edit_metadata.py.

As a rough summary for you - using your example above of clicking on Z first followed by X then Y and doing a merge. Z is the "master" version, any metadata fields that have been populated will be retained. However certain metadata fields if present in X & Y that differ from the values in Z will be merged in sequence, first the data from X then the data from Y if that has anything new.

The list of fields merged include title, authors, tags, cover, publisher, rating, series, comments and custom columns. For fields that can only contain a single value like title, then only the first value found will be used from X & Y (when Z does not have one already). For fields that can contain multiple values like tags, then a complete union is built. In addition the comments field is also compared and if it differs then it is appended to.

In terms of the book formats, the same logic applies. Any found in X & Y that already exist in Z will be deleted. If X & Y both have a new format that Z does not, the one from X will be used and Y's thrown away.


Personally I don't like certain aspects of the current behaviour and like you would want some finer control but someone has to write the code to do it. Given the complications and permutations I suspect that the only solution that would keep all of us happy is a dialog that displays the data from all the merge sources and the user picks what they want from each? The metadata sources unfortunately frequently contain utter rubbish. Poor quality covers, incrorrect publish dates, non-existent, garbage data or foreign language summaries, unrelated series, ridiculous tags etc. So I have to do a huge amount of manual editing and research to get my "Z" master exactly how I like it. Then for me when I merge I don't want anything but formats to be merged 99% of the time.

For instance I have three common reason for merging:
(1) It is a duplicate newly added (no metadata) record containing only new format(s) that I want to add to my master record.
(2) It is a duplicate newly added (no metadata) record containing format(s) that I want to merge with my master record.
(3) It is a duplicate record found at a later date where both have metadata but some differing formats to merge.

Scenario (1) is handled pretty well by the existing behaviour. Where it is not quite perfect for me is that if you automatically add tags when new records are added (like "00New") than you have to clear these out before you merge. In this situation I want to merge nothing but formats into the master.

Scenario (2) can be a very painful process. Say Z has an EPUB, and X has an EPUB and a MOBI format. You have to individually open each EPUB first to compare and decide which you want to keep (remembering which row was which EPUB if putting side by side!).

If the EPUB from Z is better, then it is as "perfect" as above - i.e. but for the tag merge of 00New. If however you prefer the EPUB from X, then you have three choices:
(2a) Remove the EPUB format from Z, then merge X into Z.
(2b) Delete Z and completely populate the metadata for X from scratch
(2c) Merge Z into X

I find (2a) is usually the desired choice as it will not overwrite any data from Z (except the 00New tag handling). The "tricky" part is the comparisons and making sure you remove the format from the right row and merge the correct order. There is no "undo" in Calibre so expect a visit to the recycle bin when you screw up.

For me (2b) is only an option if I hadn't yet setup my metadata for Z. Too much of a gamble on the work required to get the data right for X.

I used to use (2c) but it suffers from a flaw. Publish date is not merged. I suspect the reason is that the (poor imho) decision not to use nullable dates means that the application cannot determine that X does not have a publish date set yet and that it should in fact overwrite with the value from Z. I need to look into this but I wonder if the value it has would match the "Date" column for newly added records and based on this it could decide that a different value could be merged? I so wish Calibre had nullable dates - I would much rather have blanks displayed where no publish date is known.

Finally you have scenario (3) which can be a real problem. First you have to go through and decide which formats win as per (2) and decide on the remove format/merge/delete strategy. However if your titles or authors vary slightly then you can end up getting different metadata for the two rows. So for instance the comments fields may differ only by some line feeds - but Calibre will combine them together when you merge and you end up with double summaries.

The above describes the manual merging. Then you have the auto merging. I ended up writing my own program which does some "pre-processing" of all the files I am adding querying the Calibre database to divide the files into subdirectories and in the case of duplicate formats renaming them. If it is a new format of an existing Calibre book, the Calibre auto-merge works great and saves a lot of manual merging effort. However if it is a duplicate format of an existing book, it needs manual inspection to decide which wins, and you don't want Calibre to just automatically discard it. I think it would be better if the auto-merge behaviour gave an option to only auto-merge new formats, but create duplicate rows (marked with a tag perhaps) where you have the same format.

I have the same sorts of issues with the metadata download behaviour - where in a perfect world more granular control is desired so you could choose which data to overwrite. However I don't have the Python coding skills to help improve it unfortunately.

[Starson17]

Originally Posted by ldolse

the metadata itself is not merged - one book's metadata wins over the other book as far as I can tell. The winner is based on the order in which you select the books before the merge.

I wrote both Merge and AutoMerge.

Merge:
Most metadata fields (author, title, publisher, series, etc.) can only take one entry. The first selected book wins.

Tag fields are merged to keep all entries, but duplicates are removed. You can't have the same tag twice. All tags from all books are kept.

Description/comments fields are concatenated. A check is done to see if the comment/description field is identical to the first selected book so you don't get the same description five times.

AutoMerge:
This does nothing more than put the incoming book format into an existing "matching" book record if that format does not already exist for that record. "Matching" is based on exact match of author and fuzzy match of title. No metadata is added to the existing record. If a matching record is found, and that record already has the incoming format, the incoming format is not added.

I don't have any idea about two books with the same format - my guess would be the winner is also purely chosen based on the order which you selected the books.

For Merge, the first selected is kept and the others are deleted. For AutoMerge, the incoming format is ignored.

[Starson17]

Originally Posted by kiwidude

Given the complications and permutations I suspect that the only solution that would keep all of us happy is a dialog that displays the data from all the merge sources and the user picks what they want from each?

Yes. Kovid has discussed writing a fine control interface for Metadata Fetching. This should alleviate some problems for bad metadata in the first place, and it will provide a starting point for a fine control interface for Merge.

Where it is not quite perfect for me is that if you automatically add tags when new records are added (like "00New") than you have to clear these out before you merge. In this situation I want to merge nothing but formats into the master.

I can see that some might prefer to not merge the "autoadded" tag. That would be easy to change in the code, but others might want to remember that their recently added book was manually merged into an existing record. On the whole, I don't see much benefit of one over the other. It's another of those "permutations" of what different people want that would be best handled with a fine control Merge interface.

Publish date is not merged. I suspect the reason is that the (poor imho) decision not to use nullable dates means that the application cannot determine that X does not have a publish date set yet and that it should in fact overwrite with the value from Z.

Yes, that's why. If a field is filled, I don't overwrite it in Merge, and date fields are always filled.

I wonder if the value it has would match the "Date" column for newly added records and based on this it could decide that a different value could be merged?

This is a good point. When I get a chance, I'll check. Feel free to post what you find or any sample code for this change. I don't use Publish Date much, but if we can detect a "null" entry in the destination and a non-null in a source, we should overwrite.

So for instance the comments fields may differ only by some line feeds - but Calibre will combine them together when you merge and you end up with double summaries.

True, but it's unusual for anyone to edit the Comments to add only a line feed. My tests show that the text from a Fetch is consistent, so if all you've done is run a Fetch Metadata on the books, then merged, you won't get multiple summaries. They'll match.

I think it would be better if the auto-merge behaviour gave an option to only auto-merge new formats, but create duplicate rows (marked with a tag perhaps) where you have the same format.

This seems reasonable, too. We could also consider merging in metadata (which is completely ignored during AutoMerge.)

Feel free to add a Bug/Enhancement Request on this.

I have the same sorts of issues with the metadata download behaviour - where in a perfect world more granular control is desired so you could choose which data to overwrite. However I don't have the Python coding skills to help improve it unfortunately.

This is in the ToDo list (but, AFAIK, not with high priority), and I think it should be done before fine grain control for Merge. The two will be similar. It's currently just a bit beyond my PyQt skills, but we'll get there eventually.

[Starson17]

BTW, I didnt' see it in this thread (there are a lot of posts quickly - to my surprise), but ISBN is intentionally never merged. This was a conscious decision/preference by me, and when I started a thread later, to see if anyone wanted ISBN merging, it was almost universally agreed that the users did not want it. I added a notice to that effect in the warning that comes up on Merge and Safe-Merge.

Another detail during Merge: An "Unknown" author or title (in the current language) is considered to be an empty field and gets overwritten if a second selected book has a title or author that is not "Unknown".

[kiwidude]

Thanks for all the input Starson17. On the whole I think the merge as implemented does the best job it can without becoming a lot more complicated, so congrats to you on that. There are as you say so many permutations and circumstances it is only a fine grain dialog that can address everything we want.

A few weeks ago I just tweaked the source code in my version as a quick compromise to fit the behaviour I wanted for comments/tags - the nice thing about open source. With that plus my own tool I mentioned for pre-processing to handle identifying duplicates etc and it does the job. Looking into that published date is the one quirk I hadn't gotten around to investigating and only thought of that possible resolution today, though as stated I generally try to merge "formats only" into my master record.

In fact I did consider proposing a "Merge Formats only" menu option because 99% of the time that is the behaviour I want but decided to see what else the Calibre developers come up with first. Thinking out loud, the behaviour could be that if no conflicting formats are found, the merge would happen with no user interaction. If conflicts are found, a dialog pops up listing the conflicting formats in the rows, and you choose which ones you want to "win" (having the ability to launch the viewer from that dialog). That would remove a few steps from the current process and keep people like me happy who almost always want to merge into their "tags & metadata already how they like it" records.

[Starson17]

Originally Posted by kiwidude

A few weeks ago I just tweaked the source code in my version as a quick compromise to fit the behaviour I wanted for comments/tags - the nice thing about open source.

Yes, that's how I started - a feature I wanted wasn't there, so I tweaked the code for myself. Then I figured I'd share and posted my tweak, and Kovid posted - why not submit it as an enhancement. I still have a few minor tweaks that are personal.

only thought of that possible resolution today

It was good thought. I like it, if we can reliably identify a "null" date.

In fact I did consider proposing a "Merge Formats only" menu option because 99% of the time that is the behaviour I want

This shouldn't be hard - we'd just add it as a third entry to the current list of 1) regular merge and 2) safe-merge (keep others). I like it, too (in the simple form of "don't touch my metadata in the destination, just add my formats") It's basically just the AutoMerge code, without having to run the author/title matching algorithm first.

If conflicts are found, a dialog pops up listing the conflicting formats in the rows, and you choose which ones you want to "win" (having the ability to launch the viewer from that dialog).

If this was written, I think people would be asking for it on all merges, not just "Merge Formats only." This moves us towards the fine grained Merge screen - what gets merged, and what gets overwritten is controlled by the new control screen, not by the order of selection. There's not much difference between choosing which of two Series or Title to keep, and which of two pdf formats to keep.