Automatically keeping earliest date in a merge

[meghane_e]

Hi,
It's been a while since I did any tinkering on Calibre. I'm trying to merge multiple older libraries with a lot of overlapping books. From what I've found, it seems like the order should be:
1. Start with the oldest library as the base library.
2. Setup the 'Add books' task as desired:
make duplicate entries or not, ignoring copies, or copying the newest file, etc

3. Switch to the newer library. Copy books from the newer library to the base library using 'Copy to Library'.

4. Since the first two libraries I'm merging are nearly identical, the biggest piece of information I want to keep is whichever #timestamp is the earliest. So I made a custom column, #orig_entry_date, default value empty.

The most straightforward method I've found would be to do a bulk edit copy of #date or #timestamp to #orig_entry_date. While there is a #last_modified field, neither #date or #timestamp are available options in bulk edit. Is that correct behavior? What would I have to do instead?

Eventually, I would want to be able to compare the two #timestamp fields from each book and use the earliest one. I could use some pointers on getting started (a Tweak or something else, etc?). Thank you!

Thank you!!

[kovidgoyal]

pubdate is available this corresponds to the published column. timestamp is not this is the Date column. Merging in any case preserves date IIRC, though experiment to check.

[meghane_e]

Thanks for response! To clarify, I'm looking for the earliest date that the book was added to the various Calibre library databases. I thought pubdate is the date the book is published by a publisher or book author, etc. Am I mistaken about that?

[kovidgoyal]

yes pubdate is published. As I said date shouldbe preserved automatically when merging.

[meghane_e]

Sorry, but there's still some confusion. In this case, I do not care about the date a book was published by an official source (author, publishing company, etc). I only care about choosing/finding the earliest date a book record was created in my Calibre databases in merging books. Currently, the only way I know is to manually select the merge order in order to retain the earliest date.

If #date column IS the correct field to use for book creation date in the database, my problem then is that it doesn't show up in my list of available Bulk Edit Search fields. The only default date fields that appear on the list are #pubdate and #last_modified. Is #date supposed to be listed there?
Thanks

[BetterRed]

Originally Posted by meghane_e

Sorry, but there's still some confusion. In this case, I do not care about the date a book was published by an official source (author, publishing company, etc). I only care about choosing/finding the earliest date a book record was created in my Calibre databases in merging books. Currently, the only way I know is to manually select the merge order in order to retain the earliest date.

If #date column IS the correct field to use for book creation date in the database, my problem then is that it doesn't show up in my list of available Bulk Edit Search fields. The only default date fields that appear on the list are #pubdate and #last_modified. Is #date supposed to be listed there?
Thanks

First up, the '#'s are only used on custom columns, standard columns don't need them, i.e. Published/pubdate, Title/title etc.

The column you want is Date/timestamp



But IIRC it isn't available in Bulk Edit Search & Replace, and I don't remember why. But nor do I see how it would help you with your Merge issue.

BR

[theducks]

{date} does not show in Bulk mode because that would apply to all selected.

Timestamp will remain on a merge (at destination), so the direction of a merge is important.
If you use Find Duplicates: Sort the results on 'Date' column, them select the top one FIRST (the first selected is the destination)as part of the Merge. (assumes sort arrow low.High)

You DO have to do this Book by Book. But it does not take that much time per book.
(And it will remind you to never let things get that out of whack in the future )

[capink]

Originally Posted by theducks

If you use Find Duplicates: Sort the results on 'Date' column, them select the top one FIRST (the first selected is the destination)as part of the Merge. (assumes sort arrow low.High)

This is not going to work because the Find Duplicates plugin will change the sorting to be always based on the marked field. This is the only way the plugin is able to keep books of the same duplicate group next to each other. So if you try sorting before using the plugin, it will fail because the plugin will override your sort options. Also if you try sorting after the duplicates results are displayed, it will mess up you duplicates results (if you displaying all groups simultaneously), and as soon as you press next/previous group, Find Duplicates will re-override you sort options again.

@ meghane_e : If you want to control the sorting with the Find Duplicates plugin, do the following:

• Use the advanced mode of the plugin, which allows to control how each duplicate group is sorted. You can choose the sorting to be based on timestamps in ascending order so that the oldest book is always the first book in the group. For more on how to use sorting in advanced mode, refer to this post.
• After that, you can manually merge the books group by group. Another option is to use the export duplicates menu entry to generate a json file containing the duplicates group. This file can then be fed to a custom script to merge duplicate groups.
  
  I have a custom action (that works in the Action Chains plugin) that can merge all the duplicates this way automatically. It always merges the books into the first one in the group, so if you have your groups sorted as pointed above, it should do what you want.
  
  If you are interested in this, I can post it here. But be warned though, this will be a destructive action with no way to restore merged books. So, you have to test this on test library first to make sure it working as you expect it. This will be offered with no guarantees. Any thing going wrong is totally your responsibility. Also note that this merges all duplicate groups, so you have to weed out the groups (By marking them as exempt) you do not want merged before using the export feature.

[theducks]

@Capink
I run it in the only show 1 Group at a time, so sorts will work in that mode

[capink]

You are right. I checked and the sort you chose will still be intact because the calibre applies a multisort and the plugin pumps the marked field to the top of sort while keeping the previous sort columns next in the stack.

[meghane_e]

Guess it's not as straight-forward an operation as I was hoping. Basically, I've ended up with several "backup" copies of my Libraries on multiple drives and have had a couple of hardware failures over the years, etc. The result is there are several thousand overlapping entries but there are also a number of no-longer available books (due to publisher closures) that slipped through the cracks.

@BetterRed: Sorry, I should have used italics for column instead of #.

@theducks: I'll checkout the Find Duplicate plugin functionality you suggested.

@caplink: After that, you can manually merge the books group by group ... Any thing going wrong is totally your responsibility. Also note that this merges all duplicate groups, so you have to weed out the groups (By marking them as exempt) you do not want merged before using the export feature.

This might have some potential. But I'll poke around first before I come back to bug you with it. Appreciate it!

I know enough Python and some Template language to try cooking my own, but it will take me some time to figure out what's involved and where the code needs to go. So it will be another couple of weeks before I feel prepared enough to ask for more specific help. Inspiration is, of course, welcome. *sigh*

Either that or I'll have finished it manually! LOL
Thanks again, everyone!

[meghane_e]

One quick question(not solution):
I've created a custom column, #date_added, using 'Column built from other columns' set to {timestamp}. If the timestamp changes, will the #date_added column also change? Is it overkill to make a second custom column that I basically save a copy of the #date_added fields (Bulk Search/Replace copy)?
Thanks!

I guess my plan now is:
a. Start with Library A (earliest one).
b. In Library A in Control Adding books, setup Adding Actions, 'Create new record for each duplicate format'. Research 'Rules to filter added files' to see about date comparing.
c. Copy Library B to Library A.
d. At least this way, after merging two Libraries, even if the timestamps are changed, I have the originals.

[BetterRed]

Originally Posted by meghane_e

One quick question(not solution):
I've created a custom column, #date_added, using 'Column built from other columns' set to {timestamp}. If the timestamp changes, will the #date_added column also change?

Yes, columns built from other columns (commonly known as 'composite' columns) are composed at run time.

Originally Posted by meghane_e

Is it overkill to make a second custom column that I basically save a copy of the #date_added fields (Bulk Search/Replace copy)?

Your call. The Date/timestamp column is editable, however I believe it is used when sending books to e-reader devices, and if it changes it can effect what's on the reader - perhaps by creating duplicate books.

Originally Posted by meghane_e

I guess my plan now is:
a. Start with Library A (earliest one).
b. In Library A in Control Adding books, setup Adding Actions, 'Create new record for each duplicate format'. Research 'Rules to filter added files' to see about date comparing.
c. Copy Library B to Library A.
d. At least this way, after merging two Libraries, even if the timestamps are changed, I have the originals.

I gave up on trying using calibre's Merge feature when I was doing my initial library builds, there were too many exceptions. But there's been a lot of new features and plugins added since 2012, so if I was doing that today I might choose to go down a different path.

I add new books into an Intake library, then use the inter-library feature in the Find Duplicates plugin, if there is a potential duplicate I use the CalibreSpy to look at the existing book in the Target library and resolve it manually.

BR