Merging Libraries with custom columns

[Starson17]

I lost my entire library due to stupidity on my part, and I'm trying to rebuild it from old backups and partial recovered deleted copies of the library.

I need to be able to merge libraries where the merge functions like a manual merge - merging all the data from the incoming record of Library 1 into the matching existing record of Library2.

A "copy to library" with the automerge turned on can be set to insert incoming formats from Library1 into existing records of Library2, but doesn't seem to have any way to bring with it the accompanying tags or custom column data from Library1.

One option is to add the records rom Library1 to Library 2 as new records (without automerge) and then do a manual merge of duplicates, which will merge custom data and tags, but it's a very lengthy process to manually process all the records.

(As a sidenote: I did some of the original coding for the merge and automerge Calibre functions a long while ago, but I've been in hospital and away from Calibre for a long time recovering. During that period I lost the entire library and my primary backup. I'm not sure if it's possible to make the library merge work the way individual record merges work within a single library.)

Thanks for any advice.

[theducks]

Welcome back


Remember there is a tweak to copy tags

# Control how tags are applied when copying books to another library

AFAIK, the destination columns must exist (and have the same target values if: 'with a fixed permitted values'). I think that if you created the new Library with the 'use existing structure' you will be fine.

[Starson17]

Originally Posted by theducks

Welcome back

Thanks! It's been a long road.

Remember there is a tweak to copy tags

# Control how tags are applied when copying books to another library

That tweak sets whether the new tag (usually applied when a new book is added) is also applied during a CTL process if a new record is created during the CTL process. I'm trying to deal with situations where I've already got a record, but the library I'm adding from has additional metadata that I want incorporated into the destination library. In that situation, I don't create a new record and that tweak doesn't seem to apply.

AFAIK, the destination columns must exist (and have the same target values if: 'with a fixed permitted values').

They do.

I think that if you created the new Library with the 'use existing structure' you will be fine.

I did create them that way or else I added the same fields to both to make the library structures identical.

Everything works great if the destination library has no matching record. The new record created has all the source record metadata.

The problem arises when a record in the source library title/author matches a record in the destination and the source library record has extra metadata, such as tags or a custom #genre column. If the two libraries have the same formats, my only choice is to create an extra record and manually merge. I wrote the first version of that manual merge function and it does what I want it to do - it merges all the data, but it's time consuming for me to manually merge all the records I need merged.

If the two records don't have the same formats, I can use Automerge, but that function only puts the incoming format into the existing record in the destination library. It doesn't bring associated metadata from the source library to the destination record.

Basically, I'd like an option in Automerge for Copy to Library to act like the manual merge works within a single library. If the author/title matches during the CTL operation, apply the manual merge process to merge the metadata of the source with the destination. When I first wrote the automerge function (with Kovid's substantial help) it was designed for normal Adding of books, not CTL, and the treatment of metadata with the incoming books wasn't at the forefront of my mind.)

Since it's been so long since I did any work on Calibre, I thought perhaps someone else (Chaley?) had already built something like this, or perhaps there is a plugin, or perhaps it's already in the buglist as a requested option. Or perhaps someone knows how to do it using some tricks and could make recommendations.

Ideally I'd like to bring in the tags and custom column data (and the comments) from the source library to the destination library record regardless of whether the destination record has all, some or none of the same formats found in the source library record.

[kovidgoyal]

Long time no see

I suggest just bringing them in as new records and then using the duplicate finder to help with merging.

Or if you are sure that the records are identical use duplicate finder to search for dupes across both libraries and then delete the dupes from one library.

[chaley]

@Starson17: like theducks, welcome back. I hope the future is rosier than the recent past seems to have been!

Re merging: I haven't touched that part of calibre. I never use it so I can't even describe what it does. I do know that db.set_metadata will update custom columns from a metadata object, but I have no idea if abd when it is called. I can imagine merging working one of three ways: the existing record has the master metadata, the new record has the master metadata, or fields in the new record that are not in the existing record are added. The last one isn't exactly easy to do.

You seem to be saying that you want the new record to be the master. I don't see that as an option, but it should be possible to do. You would call set_metadata on the existing record, giving it the mi from the new one, possibly setting force_changes=True

[Starson17]

Originally Posted by kovidgoyal

Long time no see

It's nice to be back and find that I'm still remembered the absence wasn't exactly voluntary. You play the cards that life deals you .

I suggest just bringing them in as new records and then using the duplicate finder to help with merging.

That's probably what I'll do. It's just that it takes a long time to individually handle thousands of records. It's particularly exasperating when I know that most of the records don't have additional metadata, so it's wasted effort. I've got multiple partially recovered copies of the old library and there's no good way to identify the records that have new data.

Or if you are sure that the records are identical use duplicate finder to search for dupes across both libraries and then delete the dupes from one library.

Again, that's a relatively slow manual process. Perhaps I'll crank up the old tools I used to originally write some of the Merge and Automerge code, brush up on my coding skills and tackle an improved Automerge that will bring the metadata from Library1 and merge it with metadata from Library2. It shouldn't be that hard to do, although, as you may recall, my skills were never first rate, and they are rustier than ever. Of course, the last time I did this, I was in a worse position, never having done any Calibre coding.

Most of the code I would need is already written - Copy to Library reads the data from Library1/Library2 and Merge handles metadata merging, including custom columns exactly as I need it to be handled. It's just a matter of putting the two together. I was hoping that the Duplicate Finder code might already have implemented something like this - batch merging where you know that the records truly should be merged.

I'll check, but I'd appreciate anyone would let me know if there have been any major changes in the way the source code is being accessed and used for development. I still have my old development tools installed.

Thanks for the comments. I hope all has been going well for you and Calibre. Things can't have changed too much, as the first thread I saw was another lengthy one of those "Why can't Calibre use my personal directory structure?" threads.

[Starson17]

Originally Posted by chaley

Re merging: I haven't touched that part of calibre. I never use it so I can't even describe what it does.

I know that someone, probably Kovid, improved it after I last touched it. I thought it might have been you.

I do know that db.set_metadata will update custom columns from a metadata object, but I have no idea if abd when it is called. I can imagine merging working one of three ways: the existing record has the master metadata, the new record has the master metadata, or fields in the new record that are not in the existing record are added. The last one isn't exactly easy to do.

Options 1 and 2 are built into the Merge code, which works inside a single library. The first record selected in a Merge operation is the master, no matter how many records are being merged. The second selected has priority above all others except the first, etc.

I don't need to, nor am I interested in tackling number 3 - new field creation.

I was thinking that there would be three options for CTL Automerge operations:

1) destination is master (don't change fields in the destination, but add data from the source or fill in blank fields when a format from the source is added to a record in the destination)

2) source is master (change fields in the destination to match source data, and add data to supplement or fill in blank field data when a format from the source is added to a record in the destination), and

3) function as it currently functions - make no change to metadata in the destination record, just add the new format into the existing record according to the Add settings (overwrite or skip)

New record creation rules would remain unchanged.

I suppose this could be useful for the Find Duplicates code as well. There are times I want to just merge them all, and not have to go through all the found dupe groups.

You seem to be saying that you want the new record to be the master. I don't see that as an option, but it should be possible to do. You would call set_metadata on the existing record, giving it the mi from the new one, possibly setting force_changes=True

Usually I want the old/destination record to be the master. I just want empty fields in the destination library to be filled in if the the new record has data there when the Automerge criteria is met and an incoming format is going to be added to an existing record. For multi valued fields, like tags or #genre or description/comments, I'd like the contents of the incoming source record to be concatenated to the destination record so that the minimum amount of data is lost during the CTL merge process. All of that is handled in the Merge code now. It would be the rough equivalent of doing a CTL with automerge turned off followed by a Merge for all records that meet the Automerge criteria where the old/destination record is selected as master for the merge (option 1 above) or where the new/source record is is selected as master for the merge (option 2 above).

It's nice to hear from you

(BTW, I'm not really proposing that I write this for others. As usual, my skills are too poor to do anything other than hack together something to solve my immediate personal problem of reassembling multiple recovered databases.)

[speakingtohe]

Your name was in my brain earlier today when was writing a post saying how many people helped me personally with calibre. Once that I remember clearly was with importing records way back in 2010

One thing that may help.

The find duplicates plugin will find duplicates across libraries. You can create a temp library using the copy structure option and import x books, use the plugin with the duplicate library option, put not before marked duplicates in the search bar and you will have a list of all of the books not in your main library.



What I do at this point is use the copy to library function to copy (and delete after copy) to the main library. Then I copy the duplicates to another temp duplicates library in case I want to compare later. Then it is on to the next batch of x books. It is actually pretty fast and efficient.



Helen

[kovidgoyal]

calibre's source control system has moved from bzr to git, see the sticky post in the development section. Other than that, the code layout is mostly the same. You have two ways you can approach this:

1) Make copy to library use merging code optionally. As I recall, it already does this in some circumstances, search for auto_merged_ids in actions/copy_to_library.py for details. You might need to improve it a bit to do exactly what you need.

2) Create a simple plugin that automates merging duplicates found by Find Duplicates, or extend Find Duplicates to do that.

[chaley]

Originally Posted by kovidgoyal

calibre's source control system has moved from bzr to git, see the sticky post in the development section.

And what a PITA that has been, for me at least. None of my workflow survived, and I find GIT branching and upstream repository handling strange. But so be it ...

I found that tool SmartGit/HG helped me a lot with this transition. It is free for non-commercial use. It presents the log in a very readable threaded form, has full-file presentations of changes, presents uncommitted changes clearly, makes reversions easy, and lets you easily select what "upstream repository" to use for a given operation. My secret for happiness was discovering that if I "pull" from Kovid's master, "merge" the pulled records to my local repo, and then "sync" with my own upstream repo, then that repo is in good shape for submitting the necessary "pull request". You do the "pull request" by going to github, signing in, clicking on your upstream repo, clicking "pull requests" on the right, then clicking the green "new pull request" button.

[Starson17]

Originally Posted by speakingtohe

The find duplicates plugin will find duplicates across libraries. You can create a temp library using the copy structure option and import x books, use the plugin with the duplicate library option, put not before marked duplicates in the search bar and you will have a list of all of the books not in your main library.

Thanks for the suggestions, but my problem relates to metadata, not duplicate records. I know which books are duplicates and which are not and I have no trouble getting a single library with one record for each book I own and all the fields defined. My problem is that I have valuable metadata for the the duplicate records and sometimes that valuable metadata is in the duplicate record in Library A and sometimes it's in the duplicate record in LibraryB. If I choose to keep the duplicate records in LibraryA, I lose the data in LibraryB and vice-a-versa. What I need is a merge function.

[Starson17]

Originally Posted by kovidgoyal

calibre's source control system has moved from bzr to git, see the sticky post in the development section. Other than that, the code layout is mostly the same.

Thanks. I've got Git installed and Calibre up and running on the code. At least "Hello World" is working :-)

You have two ways you can approach this:
1) Make copy to library use merging code optionally. As I recall, it already does this in some circumstances, search for auto_merged_ids in actions/copy_to_library.py for details. You might need to improve it a bit to do exactly what you need.
2) Create a simple plugin that automates merging duplicates found by Find Duplicates, or extend Find Duplicates to do that.

Ideally, the second would probably be preferable, as it gives more control over the duplicate finding. However, I'm more familiar with the code in the CTL and automerge area, so I'll probably try that first.

[Starson17]

Originally Posted by chaley

I found that tool SmartGit/HG helped me a lot with this transition. It is free for non-commercial use.

Thanks for the suggestion. I'll take a look at SmartGit.

[BetterRed]

I may be misunderstanding what you want to do but (LibraryA is assumed to have the good formats, LibraryB is assumed to have the good metadata) ...

Could you not just move the opf file from the book in LibraryB to the book in LibraryA and rebuild the LibraryA database ?

Or if the LibraryA is big, move the books in question From LibraryA to LibraryC, move the opfs from LibraryB book folders to LibraryC book folders, rebuild LibraryC database, move books from LibraryC to LibraryA, remove (now empty) LibraryC

Or copy the format files from LibraryB to somewhere or other in case you want them some time, remove the format files from LibraryB, add the format files from LibraryA to LibraryB, remove the books from LibraryA, move the books from LibraryB to LibraryA.

BR

[BetterRed]

I now remember that I used the last one a while back, I had to adjust the custom columns so that the two library's matched.