Recovering from aborted/interrupted copy to another library

[kjdavies]

Batch copy/move to another library can be interrupted. This might be by clicking the Abort button (and @kovid recently added a safety, "Are you sure?" check to help limit accidentally aborting the copy, thank you so much!) or it could be because of a system fault (target drive is full) or failure (power cut).

When this happens, by default there doesn't seem to be much you can do about it. When aborting a transfer you do get a notice that the local copies still remain and that when you try copying again, you can set it to not copy when you find a (author, title) match. This is not bad advice, but I find it limiting.

I've devised a means of managing it myself that I think works better, but could be better done with a code change.

Manual Process

• Before copying, add a tag to all titles to be copied. I usually use something like "_copy.YYYYMMDD" (with ".diff", where 'diff' is often just a letter, if I am doing multiple moves in a day).
• Move titles to new library.
• Go to new library and remove the transient tag.

This makes the process take longer (manipulating tags in big libraries with lots of hierarchical elements in the tag browser can be slow) but it gives me a manual means of dealing with failure: go to the target library, delete all entries with the tag used by the failed copy, go back to source library and start again.

I have the sense that with a small change to the copy process, a lot of this can be done automatically (and with less user downtime waiting while the tag manipulations are processed).

• On initiating bulk copy, assign a tag as above to the list. If automating, I'd probably do something like _calibre-move.YYYYMMDD.HH24MI, using the abbreviated timestamp as adequately unique.
• Move titles as before.
• Remotely (i.e. from source library doing the copy) remove the interim tag from target objects.
• Locally, remove tag from source objects (if copying only), or delete (if moving).

This still leaves the need for manual intervention, but at least the pieces are there.

... but it made me think of a refinement that could be even better.

• Before copying, assign an identifier (calibre-operation:<operation>-YYYYMMDD.diff -- where <operation> is 'move', 'copy', etc., YYYYMMDD is obvious, and 'diff' might be the local item ID or even just order within the set).
  • Presumably should not overwrite an existing ID; if we see these IDs in the library I would expect these indicate an incomplete process and we're recovering, so don't overwrite them.
• Copy/move as usual.
• Clean up identifiers afterward, as before.

This has the advantage of making it safe to rerun, then use Find Duplicates (based on this transient ID) to clean up duplicates. Still manual, but easier to fix... but there's another option that goes a step further.

Modify the copy/move to skip entries where an identifier value is the same in both libraries. That is, if I have copied titles with calibre-op:move.20231022.3333 and calibre-op:move.20231022.3334, but got interrupted before calibre-op:move.20231022.3335 completes, I can rerun and the first two will not be copied again (ID already present) and the third will.

I can assign an identifier like this (via regex metadata update is the simplest that comes to mind) and move the files, I can still manually recover, but I think code changes to 1. not copy if the target has the title-by-ID already, and 2. automatically set and clean up the transient IDs would make this more or less seamless and faster.

Thoughts? Is there a way to do this without involving code changes? I've skimmed the Action Chains documentation and don't think AC can do this. A plugin specific to purpose might be able to, but it would feel bad to reimplement the book copy/move code (probably okay if the copy/move code as pre-operation and post-operation hooks).

Thoughts? Comments?