calibredb 'merge'

[chaley]

Originally Posted by kovidgoyal

@charles: Yes that's fine.

PR pushed

@kjdavies:
Once the above PR is accepted (it is accepted), and once it is released if you are not running from source, this script merges metadata from N books into one.

EDIT: merge_book_metadata() was moved to become a db new_api method. I changed the script.

from calibre.db.legacy import LibraryDatabase

if not LibraryDatabase.exists_at(sys.argv[1]):
    print(f'no such library or not a library: {sys.argv[1]}')
    exit()
    
db = LibraryDatabase(sys.argv[1])

ids = sys.argv[2:]
if len(ids) < 2:
    print('must supply at least 2 ids')

ids = [int(i) for i in ids]
dest_id = ids[0]
ids = ids[1:]

db.new_api.merge_book_metadata(dest_id, ids)

You run it with

calibre-debug -e path-to-script path-to-library book_id book_id [book_id *]

The first book_id is the target for the merge.

[kjdavies]

Originally Posted by chaley

PR pushed

@kjdavies:
Once the above PR is accepted (it is accepted), and once it is released if you are not running from source, this script merges metadata from N books into one.

EDIT: merge_book_metadata() was moved to become a db new_api method. I changed the script.

from calibre.db.legacy import LibraryDatabase

if not LibraryDatabase.exists_at(sys.argv[1]):
    print(f'no such library or not a library: {sys.argv[1]}')
    exit()
    
db = LibraryDatabase(sys.argv[1])

ids = sys.argv[2:]
if len(ids) < 2:
    print('must supply at least 2 ids')

ids = [int(i) for i in ids]
dest_id = ids[0]
ids = ids[1:]

db.new_api.merge_book_metadata(dest_id, ids)

You run it with

calibre-debug -e path-to-script path-to-library book_id book_id [book_id *]

The first book_id is the target for the merge.

Sweet, thanks @chaley. I'll check it out as soon as I can see it.

[kjdavies]

@chaley, I just downloaded and installed 6.28 (Windows 10).

I tried to merge two titles via the GUI and I get an error message:

calibre, version 6.28.0
ERROR: Unhandled exception: <b>AttributeError</b>:'tuple' object has no attribute 'extend'

calibre 6.28  embedded-python: True
Windows-10-10.0.19045-SP0 Windows ('64bit', 'WindowsPE')
('Windows', '10', '10.0.19045')
Python 3.10.1
Windows: ('10', '10.0.19045', 'SP0', 'Multiprocessor Free')
Interface language: None
Successfully initialized third party plugins: Action Chains (1, 18, 12) && Babelio_db (0, 8, 3) && Baen (1, 1, 0) && Barnes & Noble (1, 5, 2) && CalibreSpy (1, 0, 92) && Category Tags (0, 2, 8) && Comicvine (0, 14, 1) && Consolidate All Library Metadata (2, 0, 46) && Count Pages (1, 13, 2) && Embed Comic Metadata (1, 6, 6) && Entities Manager (1, 0, 19) && Extract ISBN (1, 6, 0) && FanFicFare (4, 28, 0) && Fantastic Fiction (1, 6, 4) && Fantastic Fiction Adults (1, 3, 0) && FictionDB (1, 4, 1) && Find Duplicates (1, 10, 8) && Generate Cover (2, 3, 3) && Goodreads (1, 7, 9) && Highlights to Obsidian (1, 4, 0) && Job Spy (1, 0, 239) && Manage Series (1, 5, 1) && Mass Search-Replace (1, 7, 4) && Open With (1, 8, 1) && Quality Check (1, 13, 6) && Reading List (1, 15, 1) && Recipe Reader (1, 0, 1) && Smashwords Metadata (1, 0, 2) && View Manager (1, 10, 2) && all recipes Recipe Reader (1, 0, 2)
Traceback (most recent call last):
  File "calibre\gui2\actions\edit_metadata.py", line 676, in merge_books
  File "calibre\gui2\actions\edit_metadata.py", line 734, in merge_metadata
  File "calibre\db\cache.py", line 85, in call_func_with_lock
  File "calibre\db\cache.py", line 3260, in merge_book_metadata
AttributeError: 'tuple' object has no attribute 'extend'

[chaley]

Originally Posted by kjdavies

@chaley, I just downloaded and installed 6.28 (Windows 10).

I tried to merge two titles via the GUI and I get an error message:

calibre, version 6.28.0
ERROR: Unhandled exception: <b>AttributeError</b>:'tuple' object has no attribute 'extend'

calibre 6.28  embedded-python: True
Windows-10-10.0.19045-SP0 Windows ('64bit', 'WindowsPE')
('Windows', '10', '10.0.19045')
Python 3.10.1
Windows: ('10', '10.0.19045', 'SP0', 'Multiprocessor Free')
Interface language: None
Successfully initialized third party plugins: Action Chains (1, 18, 12) && Babelio_db (0, 8, 3) && Baen (1, 1, 0) && Barnes & Noble (1, 5, 2) && CalibreSpy (1, 0, 92) && Category Tags (0, 2, 8) && Comicvine (0, 14, 1) && Consolidate All Library Metadata (2, 0, 46) && Count Pages (1, 13, 2) && Embed Comic Metadata (1, 6, 6) && Entities Manager (1, 0, 19) && Extract ISBN (1, 6, 0) && FanFicFare (4, 28, 0) && Fantastic Fiction (1, 6, 4) && Fantastic Fiction Adults (1, 3, 0) && FictionDB (1, 4, 1) && Find Duplicates (1, 10, 8) && Generate Cover (2, 3, 3) && Goodreads (1, 7, 9) && Highlights to Obsidian (1, 4, 0) && Job Spy (1, 0, 239) && Manage Series (1, 5, 1) && Mass Search-Replace (1, 7, 4) && Open With (1, 8, 1) && Quality Check (1, 13, 6) && Reading List (1, 15, 1) && Recipe Reader (1, 0, 1) && Smashwords Metadata (1, 0, 2) && View Manager (1, 10, 2) && all recipes Recipe Reader (1, 0, 2)
Traceback (most recent call last):
  File "calibre\gui2\actions\edit_metadata.py", line 676, in merge_books
  File "calibre\gui2\actions\edit_metadata.py", line 734, in merge_metadata
  File "calibre\db\cache.py", line 85, in call_func_with_lock
  File "calibre\db\cache.py", line 3260, in merge_book_metadata
AttributeError: 'tuple' object has no attribute 'extend'

This is a regression introduced when merge books became a calibre db API method. The exception happens when both the source and the destination of a multiple-value custom column (like tags) have values.

It is fixed in source.

[kjdavies]

Originally Posted by chaley

This is a regression introduced when merge books became a calibre db API method. The exception happens when both the source and the destination of a multiple-value custom column (like tags) have values.

It is fixed in source.

Thank you.

Does it still integrate the tags (resulting in the union of the two sets)?

[chaley]

Originally Posted by kjdavies

Thank you.

Does it still integrate the tags (resulting in the union of the two sets)?

No, the tags are not modified. As far as I can see all the other metadata is merged.

EDIT: Merging stops at the error, which means that if there are multiple source books then all metadata for some books might not be merged into the destination.

[kjdavies]

Originally Posted by chaley

No, the tags are not modified. As far as I can see all the other metadata is merged.

EDIT: Merging stops at the error, which means that if there are multiple source books then all metadata for some books might not be merged into the destination.

If I'm not mistaken -- and I've watched this happen tens of thousands of times while deduping my libraries, with tag fields involved -- when merging two titles in the library, the result contains the union of the tags.

That is, if TitleA has tag1, tag2, tag3, and TitleB has tag2, tag3, tag4, then after the merge I'm left with TitleA with title and author and cover of the original TitleA[1] and tag1, tag2, tag3, tag4.

[1] unless null -- if author is Unknown or there is no cover in TitleA, but TitleB has author or cover, the resulting TitleA will pick them up from the TitleB entry.

IOW, if merge still has the same behavior but different access, we're all good.

[kovidgoyal]

yes, th ebehavior has not changed.

[chaley]

Originally Posted by kjdavies

IOW, if merge still has the same behavior but different access, we're all good.

Originally Posted by kovidgoyal

yes, th ebehavior has not changed.

This is true once the regression fix is released. Until then, merging multiple value custom columns where both the source and destination have values will fail and no yet-to-be-processed source books will be considered.

[chaley]

The regression is fixed in calibre release 6.28.1.

[kjdavies]

Originally Posted by chaley

This is true once the regression fix is released. Until then, merging multiple value custom columns where both the source and destination have values will fail and no yet-to-be-processed source books will be considered.

That seems like such a weird edge case, I can see how it could be overlooked.

Thanks for confirming, @chaley and @kovidgoyal.

[Terisa de morgan]

Originally Posted by chaley

The regression is fixed in calibre release 6.28.1.

Thank you. Yes, it's fixed (as I was running calibre 6.28 and it has failed at the first merge I've tried, but it works running calibre 6.28.1).

[kjdavies]

Originally Posted by chaley

The regression is fixed in calibre release 6.28.1.

So I see and confirm. Thanks @chaley.