My Calibre library is missing

[RPGoldman]

I have had the same experience as this user reports.

I am on a Mac, Mac OSX, Lion.

The Calibre Library directory is now empty. Asking Calibre (new copy of 1.7.0 this afternoon) to rebuild the library gets a message that the rebuild completed successfully, but there are still no files.

This is not something I could accidentally have done, either. It would have taken considerable effort for me to have moved these files. The fact that this has happened on a Mac as well as on Windows tends to say that it's more likely to be Calibre than a bug in Windows as others have suggested.

I have looked for these files using the 'find' program and spotlight. AFAICT they are simply gone. There are no .epub files in the Trash folder, so they didn't get deleted through the finder.

I think we must consider the possibility that Calibre really has a bug, and a really bad one.

I am traveling now, but I will try to recover my library from my Time Machine backup.

[BetterRed]

Originally Posted by RPGoldman

I think we must consider the possibility that Calibre really has a bug, and a really bad one.

Given that its 4 years and 4 months between the original post to this thread, and your post today and these usage statistics http://status.calibre-ebook.com/ it seems improbable that calibre has a defect that could cause an entire library to disappear without a trace.

BTW the originator of this thread was using EXT3 on Ubuntu Linux. Windows was mentioned in respect of two other unnamed 3rd party programs that were alleged to have done similar things. And MR member itimpi mentioned Windows as the possible culprit before it became apparent that the OP wasn't running Windows.

BR

[DoctorOhh]

Moderator Notice
Since you were comparing your plight to someone who had problems 4+ years ago using ver 0.6.x of calibre I decided the two incidents have nothing in common so I moved the post to its own thread

Originally Posted by RPGoldman

The Calibre Library directory is now empty.

The calibre library being empty indicates that the metadata.db file went missing.

Originally Posted by RPGoldman

This is not something I could accidentally have done, either. It would have taken considerable effort for me to have moved these files.

Accident, corruption who can say.

Originally Posted by RPGoldman

The fact that this has happened on a Mac as well as on Windows tends to say that it's more likely to be Calibre than a bug in Windows as others have suggested.

There is zero trend as of today that indicates folks are loosing their entire physical library.

Originally Posted by RPGoldman

I have looked for these files using the 'find' program and spotlight. AFAICT they are simply gone. There are no .epub files in the Trash folder, so they didn't get deleted through the finder.

Forget about the trash, are there epub files anywhere on your hardisk?

Originally Posted by RPGoldman

I think we must consider the possibility that Calibre really has a bug, and a really bad one.

One person with a problem does not a really bad bug make.

Originally Posted by RPGoldman

I am traveling now, but I will try to recover my library from my Time Machine backup.

Good Luck.

[JSWolf]

My guess is that the library could be corrupt and the books still in place.

[RPGoldman]

Originally Posted by BetterRed

Given that its 4 years and 4 months between the original post to this thread, and your post today and these usage statistics http://status.calibre-ebook.com/ it seems improbable that calibre has a defect that could cause an entire library to disappear without a trace.

BTW the originator of this thread was using EXT3 on Ubuntu Linux. Windows was mentioned in respect of two other unnamed 3rd party programs that were alleged to have done similar things. And MR member itimpi mentioned Windows as the possible culprit before it became apparent that the OP wasn't running Windows.

BR

If you search for "calibre library disappeared" in Google, you find many hits. So I don't agree that it's improbable that Calibre has a defect. Indeed, based on google results, I draw exactly the opposite conclusion. On mobile reader alone, I see 4 results, and there are similar questions on other boards.


If I had somehow messed this up, either moving or removing the files accidentally, (a) the files should have been found using "find" or searchlight, or (b) if I had deleted the files, they should have been found in the Mac's trash (which has not been emptied in months).

Since my books are gone completely, I am inclined to believe that this was done at a level below the Mac's GUI (using the equivalent of Unix's "rm" command). I will see if I can figure out when this happened by looking at my disk backups when I return from my trip (with much of my library inaccessible )

[PeterT]

If you're comfie with command line (or the Mac has a good file find that will search everywhere) try a search for metadata.db, the main calibre database that is located in the base directory of your calibre library.

On linux at least I would use the following command to search the entire file system for it

find / -name metadata.db

If one IS found then check the directory tree shown as containing metadata.db and see what else is found there.

[BetterRed]

Originally Posted by RPGoldman

If you search for "calibre library disappeared" in Google, you find many hits. So I don't agree that it's improbable that Calibre has a defect. Indeed, based on google results, I draw exactly the opposite conclusion. On mobile reader alone, I see 4 results, and there are similar questions on other boards.


If I had somehow messed this up, either moving or removing the files accidentally, (a) the files should have been found using "find" or searchlight, or (b) if I had deleted the files, they should have been found in the Mac's trash (which has not been emptied in months).

Since my books are gone completely, I am inclined to believe that this was done at a level below the Mac's GUI (using the equivalent of Unix's "rm" command). I will see if I can figure out when this happened by looking at my disk backups when I return from my trip (with much of my library inaccessible )

IMO by tacking your problem onto the back of a 4.5 year old thread you implied that you considered the cause of your problem was the same as the what caused the original posters problem - I stand by my 'it seems improbable' assertion. And the fact that you stated that the original problem was on a Windows installation indicated to me that you had not read the thread very carefully.

Many (most ?) of the reports of the "my calibre library disappeared" I've seen were caused by users inadvertently dragging the library to another location with their file managers. I don't blame the users, I blame the IEEE, Posix should have had a nomove attribute on files and directories, like some pre-Unix/Posix file systems had.

BR

[AnotherCat]

Originally Posted by RPGoldman

If you search for "calibre library disappeared" in Google, you find many hits. So I don't agree that it's improbable that Calibre has a defect. Indeed, based on google results, I draw exactly the opposite conclusion. On mobile reader alone, I see 4 results, and there are similar questions on other boards...

...Since my books are gone completely...

I performed your exact suggested search and it seems my luck with Google is far less than your own as I didn't find many relevant hits at all. When I reviewed those they, in the main, resolved to the poster "finding" the library so the books were in fact not "gone completely" as you are claiming in your own case.

While I did not want to spend the rest of my day searching for and reviewing the few cases where the thread was not resolved with the books found, in those that was because there was no response by the poster as to the result of their performing the suggested actions by other i.e. the outcome could not be determined by reading the post.

It would seem that you are exaggerating in making the claim that many have the same problem as you i.e. that they have completely lost their books, rather than just mislaying them (usually by some action of their own).

Given that almost everyone is of the view that it is most unlikely that Calibre has lost your books (and it would seem that if it has it is an extraordinary occurrence), why do you not just restore your backup of your library and see how that goes?

[DoctorOhh]

Originally Posted by RPGoldman

If you search for "calibre library disappeared" in Google, you find many hits. So I don't agree that it's improbable that Calibre has a defect. Indeed, based on google results, I draw exactly the opposite conclusion. On mobile reader alone, I see 4 results, and there are similar questions on other boards.

While there is a first time for everything, in 5 years of using calibre and over 4.5 years actively on this forum there has never been one time calibre caused any actual file tree to disappear. There was a time during 2009 when calibre did allow users to totally clutter up existing book trees by creating the calibre library in a source directory of existing books. Things got cluttered but nothing went missing. Calibre now has precautions in place to protect new users from this happening.

In the past when a library "disappeared" it was due to the metadata.db file being accidentally deleted or corrupted. The metadata.db file contains virtually everything you actually see in the library. On occasion a user dragged and dropped it elsewhere or deleted it thinking it was redundant. There was at least one instance where some one dragged and dropped their entire tree accidentally (not unheard of in windows) to an unknown location. They were able to find out where by searching for the metadata.db file and all was well.

Originally Posted by RPGoldman

If I had somehow messed this up, either moving or removing the files accidentally, (a) the files should have been found using "find" or searchlight, or (b) if I had deleted the files, they should have been found in the Mac's trash (which has not been emptied in months).

I'm not a Mac user but I don't know if you tell calibre to remove a book if it gets placed in the trash.

Are you sure the epub files are missing?

Are you sing any Sync or cloud software?

Originally Posted by RPGoldman

Since my books are gone completely, I am inclined to believe that this was done at a level below the Mac's GUI (using the equivalent of Unix's "rm" command). I will see if I can figure out when this happened by looking at my disk backups when I return from my trip (with much of my library inaccessible )

I am curious what you find out. In the interim have a nice trip.

Originally Posted by AnotherCat

why do you not just restore your backup of your library and see how that goes?

He stated that he would attempt this when he returns from his trip.

[theducks]

I also find it hard to believe that Calibre deleted the entire folder Structure
An Uninstall leaves the:
1)Library folder(s) intact (you must manually delete them, even if you use the 'Delete (another) Library from the Library Menu (that just removes the reference).
2)Leaves the configuration (plugins) behind

An Update that should find the configuration does not even attempt to create a new library

As mentioned: Quite a while back interlocks were created that prevented creation of a Library in a folder that was occupied and did not have a metadata.db (Calibre will use that

valid Library folder, even in the absence of a missing configuration)

The only (and it is non-calibre) thing. Something hoarked the FULL user rights to the folder/contents. The odd thing is metadata.db should still be there BECAUSE of a right to delete problem

[AnotherCat]

Originally Posted by AnotherCat

...why do you not just restore your backup of your library and see how that goes?

Originally Posted by DoctorOhh

He stated that he would attempt this when he returns from his trip.

Yes, I meant for them to do that instead of in the meantime keeping on pursuing their insistence, against the tide of advice, that their books have been deleted by Calibre.

[RPGoldman]

Many of these responses seem to assume that my library has simply been moved.

I assure you that this is not the case. As I said, I used both Unix "find" and the Mac's Spotlight file search programs. The epub files are gone.

theducks seems not to believe that the files were deleted. I assure you that this was the case.

I find on return that my backup disk has a copy as of late August with all the books. Sometime after that all the books are gone, although the metadata files remain.

This might relate to the posting of theducks saying that metadata.db should remain. That is, in fact, the case. The "Calibre Library" directory remains. It contains the metadata.db and .json files, but all of the books have been removed.

Similarly, wrt PeterT's response, the books are gone; the metadata.db file remains. And to JSWolf, no, the files are gone completely.

So my case is the exact opposite of what DoctorOhh conjectures: for me the metadata remains (although it contains no entries), but the books have all been deleted.

I stress that the books were deleted completely, rather than moved to the Mac's Trash folder, or located elsewhere on the hard drive, indicating that they were not removed by, e.g., my inadvertently dragging and dropping, or accidentally hitting the Cmd-backspace key (although note that my doing that could not have removed only the books while allowing the metadata to remain; the only easy way to do that would have been to somehow select all and only the books, which would be difficult to do even intentionally).

Thanks for your interest.

[theducks]

I don't "Believe" anything.
I have "suspicions" based from experience. Some prove false .


The facts supplied by you:
1)Metadata.db exists
2?)There are: a)no Author folders, or b) Book folders within, or c)formats within those
3)The Gui reflects 2b. This sort of indicates a valid Library action.


Do you have some sort of cloud or mirror configured?
do you have some sort of scheduled script against the Library folder

Do ALL these books have a tag News

[At_Libitum]

FWIW, the presence of a metadata.db does not say a whole lot.

If Calibre assumes a library exists at a certain location, and on launch finds there is nothing there, it will simply create a blank metadata.db.

So if something happened to your library between closing Calibre and reopening it, that caused all of the epubs to go awol including the metadata.db, then anything found after relaunching Calibre is trash fodder. Can't nothing be done with no more. Only option is having a backup or a tremendous heap of luck trying to recover with a recovery program if files were deleted (leaving in the middle how they might have gotten deleted)

From the sound of it the complete folder structure was still there, just no more epub files. Any covers or opf files still present? .original_epub's even? If these are still there ... You the only one with access to that computer?

[BetterRed]

Assume all the books in a library only have one format - lets say ePUB.

Select all books, now select Remove Books->Remove files of a specific format from selected books, select ePUB, select OK - result all ePUBs gone. The author and book folder would remain as is with their cover.jpg and metadata.opf files intact, and the metadata.db file would also remain as it was (with the 'formats' column updated).

If I wanted to do it on Windows, I would do a ctrl/a (select all books), ctrl/alt/r (get the Remove files of... dialogue, space to select ePUB, enter to perform action.

I can't recall if calibre has an option to do hard deletes rather than soft deletes, but if it does and it was set then the ePUBs would be gone 'forever'.

BR