My Calibre library is missing

[BetterRed]

But there's more...

I just deleted all the PDFs in my test library - and they all went to the recycle bin, but I can't restore them to where I had them, because they were moved to another place before they were deleted and that other place has gone.

The other regulars will recognise those 'funny' folder names from other threads relating to file name lengths.

So I think there is a real problem, on Windows at least, maybe others can do some tests to confirm they're see what I'm seeing - including linux & os/x.

I have posted at launchpad to draw the issue to Kovid Goyal's attention - https://bugs.launchpad.net/calibre/+bug/1242864

@RPGoldman - if your original post had been less belligerent, and not tacked onto the back of an ancient thread, then we may have reached this point sooner.

BR

[theducks]

Originally Posted by BetterRed

But there's more...

I just deleted all the PDFs in my test library - and they all went to the recycle bin, but I can't restore them to where I had them, because they were moved to another place before they were deleted and that other place has gone.

The other regulars will recognise those 'funny' folder names from other threads relating to file name lengths.

So I think there is a real problem, on Windows at least, maybe others can do some tests to confirm they're see what I'm seeing - including linux & os/x.

I'll post something at launchpad to draw the issue to Kovid Goyal's attention.

@RPGoldman - if your original post had been less belligerent, and not tacked onto the back of an ancient thread, then we may have reached this point sooner.

BR

BR you can't use restore anyway.
Calibre deletes the (pdf) DB entry associated with those books.
Restore would only put the file back. The DB does not know about that

[BetterRed]

Originally Posted by theducks

BR you can't use restore anyway.
Calibre deletes the (pdf) DB entry associated with those books.
Restore would only put the file back. The DB does not know about that

You could fix that by opening the book folder and dragging the file to book details. I'm sure I've done it many times in the past. And the database check would detect the unrecorded formats

Now when I do a restore I get what you see in the attachment, so I must open each of those 'funny' folders - find the book in calibre drop the format on book details and then delete the 'funny' folder. That will restore its presence in the recycle bin, and then when I see it again tomorrow, I'll be scratching my head and mumbling expletives.

BR

[BetterRed]

Followers of this thread might want to pick it up at the launchpad link I previously posted.

The delete to 'funny folders' in the Recycle Bin, is a feature of the new database backend. I have made a some suggestions and observations at launchpad.

BR

[kovidgoyal]

Originally Posted by BetterRed

Now when I do a restore I get what you see in the attachment, so I must open each of those 'funny' folders - find the book in calibre drop the format on book details and then delete the 'funny' folder. That will restore its presence in the recycle bin, and then when I see it again tomorrow, I'll be scratching my head and mumbling expletives.

So dont do a restore, instead drag and drop the file you are restoring from the recycle bin into calibre.

And just as counterpoint, in the old scheme, try deleting an an entire book, then do a restore. The book folder will go back into the library, but calibre will not know what it is, therefore a maintenance check will only allow you to delete it, not restore it. SO in order to actually restore it, you will need ot

1) find it in the calibre library folder,
2) move it out, (since calibre will not allow you to add fromwithin the library folder)
3) add it to calibre via the add books from sub-folders (one per folder) option
4) delete the original

With the new scheme step 2 is no longer necessary.

[kovidgoyal]

And even with restoring individual files, just think about what happens if you

1) delete an individual format
2) Happen to change the title or author of the book
3) click restore in the recycle bin
4) Voila! Files that calibre does not know what to do with

The most important point about a calibre library is that you should not operate on the files and folders in it using anything other than calibre. That includes the recycle bin. The new db backend ensures that you cannot use the recycle bin to mess up the calibre library file structure. That is a good thing.

[RPGoldman]

Originally Posted by At_Libitum

FWIW, the presence of a metadata.db does not say a whole lot.

If Calibre assumes a library exists at a certain location, and on launch finds there is nothing there, it will simply create a blank metadata.db.

So if something happened to your library between closing Calibre and reopening it, that caused all of the epubs to go awol including the metadata.db, then anything found after relaunching Calibre is trash fodder. Can't nothing be done with no more. Only option is having a backup or a tremendous heap of luck trying to recover with a recovery program if files were deleted (leaving in the middle how they might have gotten deleted)

From the sound of it the complete folder structure was still there, just no more epub files. Any covers or opf files still present? .original_epub's even? If these are still there ... You the only one with access to that computer?

Sorry, my earlier post was not clear: no, the complete folder structure is not there. The only thing that remained was the two metadata.* files. Not only the epub files, but all of the nested folders are gone.

[RPGoldman]

Originally Posted by AnotherCat

IGiven that almost everyone is of the view that it is most unlikely that Calibre has lost your books (and it would seem that if it has it is an extraordinary occurrence), why do you not just restore your backup of your library and see how that goes?

I have restored all the epub files from my Time Machine backup, and all seems to be well. Meta-data may have been lost, but I won't worry about that.

If there's anything I can do to help detect whether this could have been done by Calibre, I will do so. Unfortunately, I have no idea how to replicate this.

[RPGoldman]

Originally Posted by theducks

I don't "Believe" anything.
I have "suspicions" based from experience. Some prove false .


The facts supplied by you:
1)Metadata.db exists
2?)There are: a)no Author folders, or b) Book folders within, or c)formats within those
3)The Gui reflects 2b. This sort of indicates a valid Library action.


Do you have some sort of cloud or mirror configured?
do you have some sort of scheduled script against the Library folder

Do ALL these books have a tag News

I don't have any cloud or mirror configured, nor any scheduled scripts.

The only scheduled scripts that I know of that would touch this are:

1. Apple's "searchlight" indexing program
2. Apple's periodic Time Machine backup system

A quick eyeball scan in Calibre shows no News tag, in the tags display column, but I don't know whether any such tag would have been lost.

[RPGoldman]

Originally Posted by BetterRed

Assume all the books in a library only have one format - lets say ePUB.

Select all books, now select Remove Books->Remove files of a specific format from selected books, select ePUB, select OK - result all ePUBs gone. The author and book folder would remain as is with their cover.jpg and metadata.opf files intact, and the metadata.db file would also remain as it was (with the 'formats' column updated).

If I wanted to do it on Windows, I would do a ctrl/a (select all books), ctrl/alt/r (get the Remove files of... dialogue, space to select ePUB, enter to perform action.

I can't recall if calibre has an option to do hard deletes rather than soft deletes, but if it does and it was set then the ePUBs would be gone 'forever'.

BR

OK. Thanks. Since the sub-folders are all gone, as well as the .ePub files, I don't believe this can be the explanation.

[RPGoldman]

I see in my Calibre library that some of my files are MOBI, some ePub, some AZW, and some PDF.

So I think this rules out my selecting all of some type.

I suppose this still leaves open the possibility that somehow I selected all the books in the library and deleted them? But I would have had to acknowledge a modal dialog box that would have checked for confirmation, right?

[theducks]

Originally Posted by RPGoldman

I don't have any cloud or mirror configured, nor any scheduled scripts.

The only scheduled scripts that I know of that would touch this are:

1. Apple's "searchlight" indexing program
2. Apple's periodic Time Machine backup system

A quick eyeball scan in Calibre shows no News tag, in the tags display column, but I don't know whether any such tag would have been lost.

The only reason I asked: IIRC News (tagged) is the only title that could get a Auto-delete if other options were also enabled

On the folder structure:

That kind of 'empty' is just a brand new library
Calibre will create a NEW Library if the referenced one is not found
Calibre will NOT create a library using an existing (non-empty) folder. (Another way of saying that it will not stomp on existing books/files)
Something else deleted that folder and Calibre just did its normal job on startup

[BetterRed]

Originally Posted by RPGoldman

I suppose this still leaves open the possibility that somehow I selected all the books in the library and deleted them? But I would have had to acknowledge a modal dialog box that would have checked for confirmation, right?

Not if you had unticked the 'Show this warning again' box - see attachment

I've never used calibre on OS/X but on Windows I sometimes inadvertently tap 'ctrl/a' (select all books) rather than a 'alt/a' (show books for current author). And then I reach for something to undo what i just did - hitting the 'del' key would not be beyond the bounds of possibilities - more so if I was stressed, in a hurry, a little tired and emotional...

Preferences->Behaviour has a button to reset the disabled confirmation dialogues.

BR