Calibre deleting files after edit metadata

[JimmXinu]

Originally Posted by JSWolf

Do you think this could be related to the newly added locking in Windows added to 6.20/6.21?

I assumed so. That's why I brought it up.

However, I'm trying to duplicate it using v6.20.0 portable and I'm not immediately able to.

[JimmXinu]

But I still see it happen with v6.22.0 regular install.

And I see the OP's issue with the v6.20.0 portable if I use the same library & book instead of a different one.

v6.19.1 portable does not have the issue.

It's possible this is tied some how to a particular library/story?

[ownedbycats]

Library paths? Windows can be weird about the user folders, so I'm thinking something like that.

[Comfy.n]

Originally Posted by ownedbycats

Library paths? Windows can be weird about the user folders, so I'm thinking something like that.

I've tried a couple of times to use my library in the user folder, but as I encountered frequent issues that would never happen with the library under non-special folders, I had to revert to c:\libpath

Thankfully, until now this topic's issue never happened to me, however I also rarely use windows explorer for searches and have search indexing turned off.

[ownedbycats]

I use XYplorer and Everything. I ignore the user folders pretty much entirely.

[BetterRed]

I use Windows Search from Windows File Explorer 'a lot', sometimes dozens of times a day. Calibre is usually running because if the search finds files in a calibre library I use the Drop Search Results plugin to "Mark the books", and/or create "Reading Lists".

I suspect this is something to do with the recent folder locking changes. Here's why:

1. I just did a search for a word (Gorbachev) I knew would be found in some of the DOCX, PDF and EPUB format files in my Books library (there were 53 of them).
   
2. With the Windows search results widow open and visible, I located and selected the first book in the list (Book#1168, Garton Ash's "History of the Present").
   
3. Pressed F2 on the Title cell and added "in September" to the title, saw the file name in the Windows Search results window change accordingly.
   
4. Did a Library Check -- no problems.
   
5. Opened the book folder, pressed F2 on the Title cell removed "in September', and saw the Book folder name and contents change accordingly, as did the Search results window.
   
6. Did a Library check -- no problems.
   
7. Added a 'data' folder to the Garton Ash book and copied James Baker's "Why Gorbachev Mattered" essay onto it and repeated the above - with no problems - the 'data' folder and its content were preserved
   
   Bonus, Windows Search found the Baker essay in Book 1168's 'data' folder.

The folder locking changes were introduced in version 6.20. for an unrelated reason I have decided to freeze my calibre installation at version 6.18.1.

BR

[BetterRed]

I repeated the above with the X1 and FileLocator (aka as Agent Ransack) search tools with same results, they detected changes to the book folder and file names as they happened and reacted accordingly, and the integrity of the calibre library was preserved.

BR

[kovidgoyal]

@JimmXinu if windows explorer is locking folders after a search completes, that is clearly a bug in windows explorer.

[ownedbycats]

I ran into an issue when using FanFicFare to update a story that had a data folder.

Traceback (most recent call last):
  File "calibre_plugins.fanficfare_plugin.fff_plugin", line 1798, in update_books_loop
  File "calibre_plugins.fanficfare_plugin.fff_plugin", line 2268, in update_metadata
  File "calibre\db\legacy.py", line 588, in set_metadata
  File "calibre\db\cache.py", line 85, in call_func_with_lock
  File "calibre\db\cache.py", line 1691, in set_metadata
  File "calibre\db\cache.py", line 1686, in set_field
  File "calibre\db\cache.py", line 1481, in set_field
  File "calibre\db\backend.py", line 1726, in windows_check_if_files_in_use
  File "calibre\utils\copy_files.py", line 249, in windows_check_if_files_in_use
  File "calibre\utils\copy_files.py", line 231, in register_folder_recursively
  File "os.py", line 225, in makedirs
PermissionError: [WinError 5] Access is denied: 'C:\\WINDOWS\\system32\\data'

Neither book nor folder was open. Windows' indexing is outright disabled, but I used Everything to search for the book earlier. (There were no open file handles when I checked though.)

Additionally, the system32 folder is a bit weird. I don't think is a data folder in there.

The FanFicFare author is equally baffled:

Originally Posted by JimmXinu

An error at that location should have the opposite effect you describe--updated epub but not metadata. Unless Calibre didn't roll back the update, in which case, you might still see metadata update.

That's Calibre code and the 'data' dir is a Calibre feature. Kovid has decided Calibre needs to be a lot more assertive about windows file locking. I don't know enough about it to say whether he's correct or not.

After the third time I bumped into the problem I reported I went back to Calibre v6.19.1. I was seeing it happen with file explorer 'Quick access', not just searches.

I haven't done anything with data directories, but judging by the system32 directory, I would suspect it's something somewhere looking for a relative 'data' without a full path.

[chaley]

Originally Posted by ownedbycats

I ran into an issue when using FanFicFare to update a story that had a data folder.

Traceback (most recent call last):
  File "calibre_plugins.fanficfare_plugin.fff_plugin", line 1798, in update_books_loop
  File "calibre_plugins.fanficfare_plugin.fff_plugin", line 2268, in update_metadata
  File "calibre\db\legacy.py", line 588, in set_metadata
  File "calibre\db\cache.py", line 85, in call_func_with_lock
  File "calibre\db\cache.py", line 1691, in set_metadata
  File "calibre\db\cache.py", line 1686, in set_field
  File "calibre\db\cache.py", line 1481, in set_field
  File "calibre\db\backend.py", line 1726, in windows_check_if_files_in_use
  File "calibre\utils\copy_files.py", line 249, in windows_check_if_files_in_use
  File "calibre\utils\copy_files.py", line 231, in register_folder_recursively
  File "os.py", line 225, in makedirs
PermissionError: [WinError 5] Access is denied: 'C:\\WINDOWS\\system32\\data'

Neither book nor folder was open. Windows' indexing is outright disabled, but I used Everything to search for the book earlier. (There were no open file handles when I checked though.)

Additionally, the system32 folder is a bit weird. I don't think is a data folder in there.

The FanFicFare author is equally baffled:

This is probably another manifestation of the problem fixed by https://github.com/kovidgoyal/calibr...0deac57851e0a6

[ownedbycats]

Should it work when running from source, or do I need to wait for update?

[chaley]

Originally Posted by ownedbycats

Should it work when running from source so I can test? I know some other similar changes required waiting for the release.

This change doesn't require a new binary. I am running it with the 6.22 binary.

FWIW: changes that require new binaries are very rare.

[ownedbycats]

That fixed it!

[OldYamaha]

So today there is a posting for an update to 6.23 that lists a bugfix for Windows
Bug fixes
Windows: Fix empty data folder getting created in directory calibre is launched from when changing title/author for a book with an existing data folder...

Is this a fix for my original post?

I went back to 6.19 and haven't had any issues with the original post since, and I don't want to update to 6.23 if this issue is still there.

Harry

Originally Posted by OldYamaha

These started last night and are now affecting my 2nd windows 10/calibre 6.21.

Here are my errors and codes:
I tried to change the title name for a book (added another word to the title "The"
Error # 1
<ERROR: permission denied
<Could not change the on disk location of this book. Is it open in another program?(No it is not, freshly opened book, has not been opened this morning after a fresh reboot in windows 10).>

copy of error

<<Locked file: C:\Users\Harry\Documents\Calibre Library\Diane Capri\07 Hunt for Jack Reacher - Deep Cov (1429)

Traceback (most recent call last):
File "calibre\utils\copy_files.py", line 95, in _open_file
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\Harry\\Documents\\Calibre Library\\Diane Capri\\07 Hunt for Jack Reacher - Deep Cov (1429)'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "calibre\utils\copy_files.py", line 95, in _open_file
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\Harry\\Documents\\Calibre Library\\Diane Capri\\07 Hunt for Jack Reacher - Deep Cov (1429)'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "calibre\gui2\metadata\single.py", line 617, in apply_changes
File "calibre\gui2\metadata\basic_widgets.py", line 241, in commit
File "calibre\db\legacy.py", line 934, in func
File "calibre\db\cache.py", line 85, in call_func_with_lock
File "calibre\db\cache.py", line 1508, in set_field
File "calibre\db\cache.py", line 1522, in update_path
File "calibre\db\backend.py", line 1875, in update_path
File "calibre\utils\copy_files.py", line 225, in copy_tree
File "calibre\utils\copy_files.py", line 120, in __enter__
File "calibre\utils\copy_files.py", line 107, in _open_file
File "calibre\utils\copy_files.py", line 111, in _open_file
PermissionError: [Errno 13] File is open in another program: 'C:\\Users\\Harry\\Documents\\Calibre Library\\Diane Capri\\07 Hunt for Jack Reacher - Deep Cov (1429)'>>

Then I click OK to exit, xcancel this error and I get this:
<you will lose all unsaved changes. Are you sure? so I click YES.>
Then I get this:

<<ERROR: Unhandled error
!Failed to read book data from calibre library. Click "Show detailss" for more information> I click show details and copied this:
<calibre, version 6.20.0
ERROR: Unhandled error: Failed to read book data from calibre library. Click "Show details" for more information

Traceback (most recent call last):
File "calibre\db\backend.py", line 1574, in copy_cover_to
PermissionError: [Errno 13] Permission denied: 'C:\\Users\\Harry\\Documents\\Calibre Library\\Diane Capri\\07 Hunt for Jack Reacher - Deep Cov (1429)\\cover.jpg'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "calibre\db\backend.py", line 1579, in copy_cover_to
PermissionError: [Errno 13] Permission denied: 'C:\\Users\\Harry\\Documents\\Calibre Library\\Diane Capri\\07 Hunt for Jack Reacher - Deep Cov (1429)\\cover.jpg'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "calibre\gui2\library\models.py", line 643, in current_changed
File "calibre\gui2\library\models.py", line 620, in get_book_display_info
File "calibre\gui2\library\models.py", line 815, in cover
File "calibre\db\legacy.py", line 455, in cover
File "calibre\db\cache.py", line 1028, in cover
File "calibre\db\cache.py", line 85, in call_func_with_lock
File "calibre\db\cache.py", line 1063, in copy_cover_to
File "calibre\db\backend.py", line 1582, in copy_cover_to
Exception: Failed to open 'C:\\Users\\Harry\\Documents\\Calibre Library\\Diane Capri\\07 Hunt for Jack Reacher - Deep Cov (1429)\\cover.jpg' with error: [Errno 13] Permission denied: 'C:\\Users\\Harry\\Documents\\Calibre Library\\Diane Capri\\07 Hunt for Jack Reacher - Deep Cov (1429)\\cover.jpg'>>

I then click OK

It creaded the new folder for the title change, but the folder is empty, and the old folder has the files that were there. the Calibre program shows the updated file name, in this case "01a Hunt for Jack Reacher-Deep Cover Jack" and will let me double click on the title in calibre to open thei file, but the file in the file folder is the old file name. And when I try to delete the entry in calibre, I get this error:

<ERROR: Permission denied
<Could not access 07 Hunt for jack reacher-deep-Diane Capri.epub. Is it being used by another program? Click "Show details" for more information. (I clicked show details and copied the error information)

calibre, version 6.20.0
ERROR: Permission denied: Could not access 07 Hunt for Jack Reacher - Deep - Diane Capri.epub. Is it being used by another program? Click "Show details" for more information.

<<Traceback (most recent call last):
File "calibre\utils\copy_files.py", line 95, in _open_file
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\Harry\\Documents\\Calibre Library\\Diane Capri\\07 Hunt for Jack Reacher - Deep Cov (1429)\\07 Hunt for Jack Reacher - Deep - Diane Capri.epub'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "calibre\utils\copy_files.py", line 95, in _open_file
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\Harry\\Documents\\Calibre Library\\Diane Capri\\07 Hunt for Jack Reacher - Deep Cov (1429)\\07 Hunt for Jack Reacher - Deep - Diane Capri.epub'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "calibre\gui2\actions\delete.py", line 459, in do_library_delete
File "calibre\gui2\library\models.py", line 466, in delete_books_by_id
File "calibre\db\cache.py", line 85, in call_func_with_lock
File "calibre\db\cache.py", line 2082, in remove_books
File "calibre\db\backend.py", line 2148, in remove_books
File "calibre\db\backend.py", line 2041, in move_book_to_trash
File "calibre\utils\copy_files.py", line 225, in copy_tree
File "calibre\utils\copy_files.py", line 118, in __enter__
File "calibre\utils\copy_files.py", line 107, in _open_file
File "calibre\utils\copy_files.py", line 111, in _open_file
PermissionError: [Errno 13] File is open in another program: 'C:\\Users\\Harry\\Documents\\Calibre Library\\Diane Capri\\07 Hunt for Jack Reacher - Deep Cov (1429)\\07 Hunt for Jack Reacher - Deep - Diane Capri.epub'>>

Now the only way to recover is to close the Calibre program, then delete the folder and files, manually restart the calibre program then delete the book.

So then I tried to clean up the Library using Library Maintenance, this did not help, has the same issue.

So then I tried to "Move" my library to the C:/ drive, out of my User folders, this did not work.

I ran Malwarebytes without finding anything, this did not do anything.

Now ALL OF A SUDDEN, BOTH PC's with Windows 10 will not let me edit ANYTHING using Calibre.

It was working ok until I upgraded to 6.21, then this started.
I uninstalled 6.21 then install 6.20, to the exact same reults as listed above.
I have a second PC that has 2 seperate operating systems, Windows 10 and Zorin Linux. They also Have 6.21 installed but DO NOT have this issue.

Could my "Calibre Library folder" be corrupted?

I also have a new folder in "My Documents" folder called .Calibre with 2 CALIBRE_DATA files 1 is 1 MB and 1 is 96,000 KB. I've never seen this type of folder, or these Data files before.

[kovidgoyal]

No, it isnt.