Calibre has deleted the whole library

[asv]

Greetings,

As "Issues" section at GitHub is disabled, this seems to be the best place to post this.

Preface:
I have been using Calibre for years on various Linux(es) and currently on Windows 10 system. Today, the application has greeted me with an empty window. Slightly surprised, I have checked the library folder which turned out to be entirely empty, both books and DB are gone. No error message was shown.

Interlude
I see some logs here, but content isn't alarming:

..\AppData\Local\calibre-ebook.com\calibre\QtWebEngine\viewer-lookup\Platform Notifications\

Next,

..\AppData\Local\calibre-cache\

\AppData\Roaming\calibre\

and

\AppData\Roaming\calibre\

all look intact.

While the

\AppData\Local\Calibre_library\

is empty.

Foreword
It was a fun ride and I am grateful to the Kovid G. and contributors for their work.
Hope my humble donations have helped at least a tiny bit.

I have archived the following folders in case developer(s) is/are reading this forum and interested in some files:

Calibre_library
calibre-cache
calibre-ebook.com
calibre-parallel

[asv]

Important bits:

Versions that has caused the data loss: v4.22
OS: Windows 10.0.18363.0 (FDE/HVCI/CG enabled)
Installed plugins: "Find duplicates" v1.6.3

[asv]

UPDATE
As it turns out, library content was moved to "Recycle bin", not deleted permanently.

However, these files were zeroed (0 bytes):

metadata.db
metadata_db_prefs_backup.json

Restoring metadata from a backup...

[asv]

Peculiar fact:
Restoring metadata.db and moving the books back doesn't bring the database to life. Main Calibre window is empty after the application restart.

[stumped]

i would blame that out of date looking windows 10 you are running and/or its associated windows defender AV, not calibre.

assuming its not user error, it looks like an over zealous AV or reg. cleaner or some such has removed what it thought was junk / malware.

NB if you are expecting to find your library books in \AppData\Local\Calibre_library.. you are looking in the wrong place, unless you have a very strange non standard install. your books sh\uld be in documents/calibre library. NB If you have allowed microsoft One drive to sync that folder, which it now tries to do by default, you have a new leading suspect...

Create an empty new library then use " add books" to add all the books back again from a backup folder, that should give you a working library

[JSWolf]

Originally Posted by stumped

i would blame that out of date looking windows 10 you are running and/or its associated windows defender AV, not calibre.

assuming its not user error, it looks like an over zealous AV or reg. cleaner or some such has removed what it thought was junk / malware.

NB if you are expecting to find your library books in \AppData\Local\Calibre_library.. you are looking in the wrong place, unless you have a very strange non standard install. your books sh\uld be in documents/calibre library. NB If you have allowed microsoft One drive to sync that folder, which it now tries to do by default, you have a new leading suspect...

Create an empty new library then use " add books" to add all the books back again from a backup folder, that should give you a working library

It's not Windows 10 (even though it's not the current version) or Windows Security. I cannot say what it is, just what it isn't.

[asv]

Originally Posted by stumped

i would blame that out of date looking windows 10 you are running and/or its associated windows defender AV, not calibre.

1909 is supported until May 2021 for Home/Pro/WS/IoT editions.

Originally Posted by stumped

assuming its not user error, it looks like an over zealous AV or reg. cleaner or some such has removed what it thought was junk / malware.

Not using any kind of "reg. cleaners" and Windows Defender log has nothing related to the library:

> Get-MpThreatDetection | Select-Object -Property Resources | grep -i lib
>

Originally Posted by stumped

NB if you are expecting to find your library books in \AppData\Local\Calibre_library.. you are looking in the wrong place, unless you have a very strange non standard install.

This is indeed where I store my books.

Originally Posted by stumped

NB If you have allowed microsoft One drive to sync that folder, which it now tries to do by default, you have a new leading suspect...

Fortunately, uninstalling OneDrive was amongst the first steps I have taken after installing the OS

Originally Posted by stumped

Create an empty new library then use " add books" to add all the books back again from a backup folder, that should give you a working library

I have managed to restore metadata.db from a backup, is there a "community-approved" way to merge it back? I'd like to have proper meta for PDFs, etc.

[stumped]

Originally Posted by JSWolf

It's not Windows 10 (even though it's not the current version) or Windows Security. I cannot say what it is, just what it isn't.

if the library books really were deliberately kept in.. appdata/local/calibre_library
then many utilities / security suites will monitor/clean that if allowed. Because Its a prime location for ransomware to download to & launch from. Perhaps OP can elaborate on where/why his library was set up to be?

( out of curioslty, I inspected my appdata/local for a possible calibre_library folder, I have no such entry )

[theducks]

And using Windows recovery tools, will clean %appdata%
IMHO a very 'Not Safe' location
(I just used the tool to 'Reset' recently. Programs/drivers that have been running under W10 for years, are not even allowed to install. Appdata was wiped (moved to a 'old') of all non-windows entries )

[hobnail]

Windows also has a thing where it periodically runs and removes what it considers junk/dead files. I was stupidly storing stuff in my windows temp folder and one day found them gone after they'd been there awhile untouched.

[stumped]

Originally Posted by asv

1909 is supported until May 2021 for Home/Pro/WS/IoT editions.
..
.... It was not clear from #5 that you have 1909. The number therein began with 18....

I have managed to restore metadata.db from a backup, is there a "community-approved" way to merge it back? I'd like to have proper meta for PDFs, etc.

Metadata for each book is backed up in opf files stored alongside the books, if you still have those.

I think one of the library management repair tools will use those to rebuild what is in metadata.db, but it's not something I have ever tried to do

[BetterRed]

Originally Posted by stumped

NB if you are expecting to find your library books in \AppData\Local\Calibre_library.. you are looking in the wrong place, unless you have a very strange non standard install. your books sh\uld be in documents/calibre library.

FTR: Following MS's introduction of Controlled Folder Access (CFA) as an anti-ransomware measure in 2017 the default location for Calibre's default library was changed from %USERHOME%\Documents to %USERHOME% (e.g. C:/Users/BetterRed/Calibre Library).

See ==>> Prevent ransomware and threats from encrypting and changing files - Windows security | Microsoft Docs.

If 'CFA' is enabled the %USERHOME%\Documents folder gets "Controlled" by default.

BR

[kovidgoyal]

calibre has no code to delete entire libraries, in fact even when you actually ask it to remove a library it does not delete it, simply removes the listing from inside calibre.

[ownedbycats]

I seem to recall that certain updates for Windows 10 had problems with losing user data. I wonder if that might've happened here. Or some antivirus getting angry.