Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Library Management

Notices

Reply
 
Thread Tools Search this Thread
Old 04-12-2021, 05:47 PM   #1
meghane_e
Connoisseur
meghane_e began at the beginning.
 
Posts: 73
Karma: 10
Join Date: Sep 2016
Device: Kindle
Automatically keeping earliest date in a merge

Hi,
It's been a while since I did any tinkering on Calibre. I'm trying to merge multiple older libraries with a lot of overlapping books. From what I've found, it seems like the order should be:
1. Start with the oldest library as the base library.
2. Setup the 'Add books' task as desired:
make duplicate entries or not, ignoring copies, or copying the newest file, etc

3. Switch to the newer library. Copy books from the newer library to the base library using 'Copy to Library'.

4. Since the first two libraries I'm merging are nearly identical, the biggest piece of information I want to keep is whichever #timestamp is the earliest. So I made a custom column, #orig_entry_date, default value empty.

The most straightforward method I've found would be to do a bulk edit copy of #date or #timestamp to #orig_entry_date. While there is a #last_modified field, neither #date or #timestamp are available options in bulk edit. Is that correct behavior? What would I have to do instead?

Eventually, I would want to be able to compare the two #timestamp fields from each book and use the earliest one. I could use some pointers on getting started (a Tweak or something else, etc?). Thank you!

Thank you!!
meghane_e is offline   Reply With Quote
Old 04-12-2021, 09:15 PM   #2
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,778
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
pubdate is available this corresponds to the published column. timestamp is not this is the Date column. Merging in any case preserves date IIRC, though experiment to check.
kovidgoyal is offline   Reply With Quote
Old 04-15-2021, 03:33 AM   #3
meghane_e
Connoisseur
meghane_e began at the beginning.
 
Posts: 73
Karma: 10
Join Date: Sep 2016
Device: Kindle
Thanks for response! To clarify, I'm looking for the earliest date that the book was added to the various Calibre library databases. I thought pubdate is the date the book is published by a publisher or book author, etc. Am I mistaken about that?
meghane_e is offline   Reply With Quote
Old 04-15-2021, 03:49 AM   #4
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,778
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
yes pubdate is published. As I said date shouldbe preserved automatically when merging.
kovidgoyal is offline   Reply With Quote
Old 04-16-2021, 05:29 AM   #5
meghane_e
Connoisseur
meghane_e began at the beginning.
 
Posts: 73
Karma: 10
Join Date: Sep 2016
Device: Kindle
Sorry, but there's still some confusion. In this case, I do not care about the date a book was published by an official source (author, publishing company, etc). I only care about choosing/finding the earliest date a book record was created in my Calibre databases in merging books. Currently, the only way I know is to manually select the merge order in order to retain the earliest date.

If #date column IS the correct field to use for book creation date in the database, my problem then is that it doesn't show up in my list of available Bulk Edit Search fields. The only default date fields that appear on the list are #pubdate and #last_modified. Is #date supposed to be listed there?
Thanks
meghane_e is offline   Reply With Quote
Old 04-16-2021, 07:41 AM   #6
BetterRed
null operator (he/him)
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 20,459
Karma: 26645808
Join Date: Mar 2012
Location: Sydney Australia
Device: none
Quote:
Originally Posted by meghane_e View Post
Sorry, but there's still some confusion. In this case, I do not care about the date a book was published by an official source (author, publishing company, etc). I only care about choosing/finding the earliest date a book record was created in my Calibre databases in merging books. Currently, the only way I know is to manually select the merge order in order to retain the earliest date.

If #date column IS the correct field to use for book creation date in the database, my problem then is that it doesn't show up in my list of available Bulk Edit Search fields. The only default date fields that appear on the list are #pubdate and #last_modified. Is #date supposed to be listed there?
Thanks
First up, the '#'s are only used on custom columns, standard columns don't need them, i.e. Published/pubdate, Title/title etc.

The column you want is Date/timestamp

Click image for larger version

Name:	Screenshot 2021-04-16 212621.jpg
Views:	66
Size:	105.3 KB
ID:	186642

But IIRC it isn't available in Bulk Edit Search & Replace, and I don't remember why. But nor do I see how it would help you with your Merge issue.

BR
BetterRed is offline   Reply With Quote
Old 04-16-2021, 09:53 AM   #7
theducks
Well trained by Cats
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 29,689
Karma: 54369090
Join Date: Aug 2009
Location: The Central Coast of California
Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A
{date} does not show in Bulk mode because that would apply to all selected.

Timestamp will remain on a merge (at destination), so the direction of a merge is important.
If you use Find Duplicates: Sort the results on 'Date' column, them select the top one FIRST (the first selected is the destination)as part of the Merge. (assumes sort arrow low.High)

You DO have to do this Book by Book. But it does not take that much time per book.
(And it will remind you to never let things get that out of whack in the future )
theducks is offline   Reply With Quote
Old 04-16-2021, 01:32 PM   #8
capink
Wizard
capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.
 
Posts: 1,081
Karma: 1948136
Join Date: Aug 2015
Device: Kindle
Quote:
Originally Posted by theducks View Post
If you use Find Duplicates: Sort the results on 'Date' column, them select the top one FIRST (the first selected is the destination)as part of the Merge. (assumes sort arrow low.High)
This is not going to work because the Find Duplicates plugin will change the sorting to be always based on the marked field. This is the only way the plugin is able to keep books of the same duplicate group next to each other. So if you try sorting before using the plugin, it will fail because the plugin will override your sort options. Also if you try sorting after the duplicates results are displayed, it will mess up you duplicates results (if you displaying all groups simultaneously), and as soon as you press next/previous group, Find Duplicates will re-override you sort options again.

@ meghane_e : If you want to control the sorting with the Find Duplicates plugin, do the following:
  • Use the advanced mode of the plugin, which allows to control how each duplicate group is sorted. You can choose the sorting to be based on timestamps in ascending order so that the oldest book is always the first book in the group. For more on how to use sorting in advanced mode, refer to this post.
  • After that, you can manually merge the books group by group. Another option is to use the export duplicates menu entry to generate a json file containing the duplicates group. This file can then be fed to a custom script to merge duplicate groups.

    I have a custom action (that works in the Action Chains plugin) that can merge all the duplicates this way automatically. It always merges the books into the first one in the group, so if you have your groups sorted as pointed above, it should do what you want.

    If you are interested in this, I can post it here. But be warned though, this will be a destructive action with no way to restore merged books. So, you have to test this on test library first to make sure it working as you expect it. This will be offered with no guarantees. Any thing going wrong is totally your responsibility. Also note that this merges all duplicate groups, so you have to weed out the groups (By marking them as exempt) you do not want merged before using the export feature.
capink is offline   Reply With Quote
Old 04-16-2021, 03:14 PM   #9
theducks
Well trained by Cats
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 29,689
Karma: 54369090
Join Date: Aug 2009
Location: The Central Coast of California
Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A
@Capink
I run it in the only show 1 Group at a time, so sorts will work in that mode
theducks is offline   Reply With Quote
Old 04-16-2021, 04:29 PM   #10
capink
Wizard
capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.capink ought to be getting tired of karma fortunes by now.
 
Posts: 1,081
Karma: 1948136
Join Date: Aug 2015
Device: Kindle
You are right. I checked and the sort you chose will still be intact because the calibre applies a multisort and the plugin pumps the marked field to the top of sort while keeping the previous sort columns next in the stack.
capink is offline   Reply With Quote
Old 04-17-2021, 05:32 PM   #11
meghane_e
Connoisseur
meghane_e began at the beginning.
 
Posts: 73
Karma: 10
Join Date: Sep 2016
Device: Kindle
Guess it's not as straight-forward an operation as I was hoping. Basically, I've ended up with several "backup" copies of my Libraries on multiple drives and have had a couple of hardware failures over the years, etc. The result is there are several thousand overlapping entries but there are also a number of no-longer available books (due to publisher closures) that slipped through the cracks.

@BetterRed: Sorry, I should have used italics for column instead of #.

@theducks: I'll checkout the Find Duplicate plugin functionality you suggested.

@caplink: After that, you can manually merge the books group by group ... Any thing going wrong is totally your responsibility. Also note that this merges all duplicate groups, so you have to weed out the groups (By marking them as exempt) you do not want merged before using the export feature.

This might have some potential. But I'll poke around first before I come back to bug you with it. Appreciate it!

I know enough Python and some Template language to try cooking my own, but it will take me some time to figure out what's involved and where the code needs to go. So it will be another couple of weeks before I feel prepared enough to ask for more specific help. Inspiration is, of course, welcome. *sigh*

Either that or I'll have finished it manually! LOL
Thanks again, everyone!

Last edited by meghane_e; 04-17-2021 at 05:33 PM. Reason: punctuation
meghane_e is offline   Reply With Quote
Old 04-17-2021, 06:32 PM   #12
meghane_e
Connoisseur
meghane_e began at the beginning.
 
Posts: 73
Karma: 10
Join Date: Sep 2016
Device: Kindle
One quick question(not solution):
I've created a custom column, #date_added, using 'Column built from other columns' set to {timestamp}. If the timestamp changes, will the #date_added column also change? Is it overkill to make a second custom column that I basically save a copy of the #date_added fields (Bulk Search/Replace copy)?
Thanks!

I guess my plan now is:
a. Start with Library A (earliest one).
b. In Library A in Control Adding books, setup Adding Actions, 'Create new record for each duplicate format'. Research 'Rules to filter added files' to see about date comparing.
c. Copy Library B to Library A.
d. At least this way, after merging two Libraries, even if the timestamps are changed, I have the originals.

Last edited by meghane_e; 04-17-2021 at 06:33 PM. Reason: punctuation
meghane_e is offline   Reply With Quote
Old 04-17-2021, 06:59 PM   #13
BetterRed
null operator (he/him)
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 20,459
Karma: 26645808
Join Date: Mar 2012
Location: Sydney Australia
Device: none
Quote:
Originally Posted by meghane_e View Post
One quick question(not solution):
I've created a custom column, #date_added, using 'Column built from other columns' set to {timestamp}. If the timestamp changes, will the #date_added column also change?
Yes, columns built from other columns (commonly known as 'composite' columns) are composed at run time.

Quote:
Originally Posted by meghane_e View Post
Is it overkill to make a second custom column that I basically save a copy of the #date_added fields (Bulk Search/Replace copy)?
Your call. The Date/timestamp column is editable, however I believe it is used when sending books to e-reader devices, and if it changes it can effect what's on the reader - perhaps by creating duplicate books.

Quote:
Originally Posted by meghane_e View Post
I guess my plan now is:
a. Start with Library A (earliest one).
b. In Library A in Control Adding books, setup Adding Actions, 'Create new record for each duplicate format'. Research 'Rules to filter added files' to see about date comparing.
c. Copy Library B to Library A.
d. At least this way, after merging two Libraries, even if the timestamps are changed, I have the originals.
I gave up on trying using calibre's Merge feature when I was doing my initial library builds, there were too many exceptions. But there's been a lot of new features and plugins added since 2012, so if I was doing that today I might choose to go down a different path.

I add new books into an Intake library, then use the inter-library feature in the Find Duplicates plugin, if there is a potential duplicate I use the CalibreSpy to look at the existing book in the Target library and resolve it manually.

BR
BetterRed is offline   Reply With Quote
Reply

Tags
copy date

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Automatically Setting Date field to file creation date when importing. Marc_G2 Library Management 6 07-25-2020 10:35 PM
Keeping date order? palazzo Library Management 6 11-03-2017 01:02 PM
Keeping date order on new Kindle palazzo Devices 12 04-27-2016 11:08 AM
Automatically Advance Date? CountZero Library Management 1 05-27-2013 08:47 PM
New Problem-Merge libraries and preserve date Sydney's Mom Library Management 0 05-02-2013 08:37 PM


All times are GMT -4. The time now is 02:41 AM.


MobileRead.com is a privately owned, operated and funded community.