Automatic Add deleted books NOT added [Calibre 9.5 on Win11]

[mneimeyer]

On the Automatic add page in Preferences the warning says "WARNING: Files in the above folder will be deleted after being added to calibre."

Reading the warning I thought that only books actually added would be deleted. That the rest would be left alone. I dumped a heap of books into that folder, said don't add the duplicates that it found and it looks like calibre deleted EVERYTHING in that folder after processing (again... even the ones it did not add to my Library). To make matters worse it didn't send the removed files to the Windows Recycle bin.

In this case a LOT of the dupes were false positives caused by "Title - Author" and "Author - Title" flips, "Last, First" and "First Last" flips, and books with "&" in the Title (being read as Authors) as well as actual duplicates.

Did I miss a configuration somewhere? How do I get calibre to not delete books it didn't add (for any reason) or at the very least to get it to use the Recycle Bin?

Thanks!

P.S. For whatever it's worth I very much would consider this a bug and at the very least bad UX. First, the warning is NOT clear, in my opinion, that non-added books are also deleted. Second, the files should NOT have skipped the Recycle Bin. Not in this day and age where storage is backed by NVMe and SSD with TRIM enabled by default on Win10 and 11.

I do realize this behavior might be to avoid having files linger in the import folder but an option "Remove permanently" or "Send to Recycle Bin" would be one way to handle this. Another would be to automatically create a sub-folder "Unable to Add to Library" (or whatever...) and move files to that.

[kovidgoyal]

You dont -- the automatic adding folder is for automatic adding, not book storage. It is, quite simply, a temporary staging area, typically used for automatically adding downloaded books or similar to calibre. Not sure where you got the impression otherwise.

And yes *duplicates* you choose not to add are removed as well since a duplicate presumably already exists in your library.

[mneimeyer]

No I get that. That's not what I was saying. My Calibre library exists at "B:\Library" and my Automatic Add folder is at "B:\Library AutoAdd".

Consolidating the various versions of my library that exist on multiple hard drives and backups and such I've been bringing books into my library in batches by dropping them into the AutoAdd folder.

This time I told it not to import duplicate books (many of which were not actually duplicates but name conflicts due the bad file naming I mentioned) Calibre deleted ALL of the files in my AutoAdd folder even if they had not been imported into Calibre. I know of at least seven CBZ files that had the word "Evil" in the file name and searching for ANY file on B:\ with that word in the file name failed. The CBZs did not have embedded ComicInfo.xml so it couldn't have silently renamed the files using bad metadata. Similarly searching my Recycle Bin does not find them. I have confirmed that my Recycle Bin is correctly configured for that drive and files I delete manually through Explorer are sent there.

So to reiterate, I'm not talking about the actual Calibre Library just the contents of the AutoAdd folder.

Thanks for the rapid response! I hope that clarifies what I mean!

[kovidgoyal]

Yes I am saying that automatic adding folder is a staging area. Any books that you add to calibre from it or books that you tell calibre are duplicates when adding from it and therefor it does not add, will be deleted from the automatic add area.

[mneimeyer]

The warning says "deleted after adding" and not "deleted after attempting to add." The deletion criterion as described is successful addition, not merely processing. A file that fails to import hasn't been added. So I'd argue the warning is at least incomplete.

Beyond the warning, silently deleting files that failed to process is genuinely unusual behavior in my experience. Every program or tool that does any sort of import that I've used either flags failed imports, moves them to a separate location, or just leaves them in place so giving us at least the option would make the workflow smoother.

Thanks

[Sirtel]

You can always just let it add presumed duplicates and sort them out later. I mean, if many are false positives, you'd need to sort them out anyway, so why not just add them and do the sorting in calibre?

[mneimeyer]

Originally Posted by Sirtel

why not just add them and do the sorting in calibre?

Fundamentally because I didn't realize it would delete the files I didn't import.

Ultimately it is my goal to work inside Calibre but I can do some rough bulk edits to file names in the file system pre-import with scripts so I'd originally planned to do that.

[Sirtel]

Originally Posted by mneimeyer

Fundamentally because I didn't realize it would delete the files I didn't import.

Ultimately it is my goal to work inside Calibre but I can do some rough bulk edits to file names in the file system pre-import with scripts so I'd originally planned to do that.

Have you set calibre to take its metadata from the filenames, then? Otherwise your changing file names won't matter.

[kovidgoyal]

Its not a failure to import. When a duplicate is detected you are *asked* if you want to import it and only if you say no is it deleted.

[JSWolf]

Originally Posted by mneimeyer

Fundamentally because I didn't realize it would delete the files I didn't import.

Ultimately it is my goal to work inside Calibre but I can do some rough bulk edits to file names in the file system pre-import with scripts so I'd originally planned to do that.

Unless you have calibre reading the metadata from the filename, renaming any of those eBook is irrelevant as calibre would read the metadata in the eBook.

[mneimeyer]

Originally Posted by kovidgoyal

Its not a failure to import. When a duplicate is detected you are *asked* if you want to import it and only if you say no is it deleted.

It does ask. I didn't dispute that.

The text on the Duplicate Found Dialog says "Books with the same titles as the following already exist in calibre. Select which books you want added anyway."

The text in the warning on the adding books preference page says "Warning: Files in the above folder will be deleted after being added to calibre".

This may come down to language differences perhaps but the American version of English I was raised on would say that if a book is not selected then it is not added and if a book is not added then it should not qualify to be deleted. Phrased inversely, only books that are added will be deleted.

Now that I know this I will be sure to do things the expected way but it feels like such as easy fix to just change the warning to something like "Files in the above folder will be removed after being processed whether they are added to the library or not".

I would still argue that not using the recycle bin is a bad User Experience but properly warned it becomes less so.

[mneimeyer]

Originally Posted by JSWolf

Unless you have calibre reading the metadata from the filename, renaming any of those eBook is irrelevant as calibre would read the metadata in the eBook.

CBR and CBZ files, like I was adding, that do not have a ComicInfo.xml have no metadata to read.

[JSWolf]

i agree that if a book is not added it should not be deleted. I do think this needs to be fixed because it's a bug.

[theducks]

I am going to side with Jon on this one (or it needs to be an option to Leave them behind).

When I was using auto add (a while back), I would come back to the folder and find remnants and know I needed to figure out 'Why' those failed (usually was my fault ). Depending on cases, a log file is not always the easiest way to chase things.

[JSWolf]

Originally Posted by theducks

I am going to side with Jon on this one (or it needs to be an option to Leave them behind).

When I was using auto add (a while back), I would come back to the folder and find remnants and know I needed to figure out 'Why' those failed (usually was my fault ). Depending on cases, a log file is not always the easiest way to chase things.

If an option is added, I think the option should default to not deleting if the books are not added. IMHO, it's too easy to hit the wrong button when being told this the eBook may be a duplicate and if you do that, it would be deleted.