Sudden Big Problems with Library - Page 2

eschwartz · 08-19-2015, 01:47 PM

Quote:

Originally Posted by theducks

All of a sudden, 'Google Drive' entered this discussion.

Reread post #3.

theducks · 08-19-2015, 02:25 PM

Quote:

Originally Posted by eschwartz

Reread post #3.

BranMakMorn · 08-20-2015, 04:34 AM

Well I did use the fix, but it fixed a very small amount of errors. I still find myself with several missing PDFs (luckily, I think only something like 20-30 rather than over 100). As well as a LOT of folders with two copies of the same PDF, or two copies of the metadata, or the cover file, which calibre sees in the database but tells me has no associated file. I am manually fixing the issue (deleting the extra files) for the few hundred cases I can easily identify (those titles Calibre tells me have no "size" in Mb). I've already closed Google Drive. Once I'm done with this I will uninstall, reinstall and put my library into OneDrive, hoping that in the process of reconstituting the library Calibre will "find" those files which now says are missing but are not.

As for the lost books, that's kind of a mystery because it seems to me that the total size of the Calibre folder is the same.

PeterT · 08-20-2015, 09:00 AM

There is no need to uninstall / reinstall calibre to create a new library; just right click on the icon in the toolbar that shows the library, and select the option Switch/create library... and make your new library where you want it.

BranMakMorn · 08-20-2015, 12:08 PM

OK so:

1) I moved the Calibre Library out of Google Drive, and it is now in the OneDrive folder, but I paused the sync immediately, to make sure I could fix the library via Calibre first, and sync later;

2) I reinstalled Calibre, and told it to look for a previously existing library in the new directory;

3) I did a "restore database", which took ages, but gave me back several errors. I give an example here

calibre, version 2.35.0
WARNING: Success: Restoring the database succeeded with some warnings. Click "Show details" to see the details. The old database was saved as: C:\Users\XXXXX\OneDrive\Documents\Calibre Library\metadata_pre_restore.db

Failed to restore the books in the following folders:
C:\Users\XXXXX\OneDrive\Documents\Calibre Library\Jack Doe\Your Computer Sucks (271) with error:
Traceback (most recent call last):
File "site-packages\calibre\db\restore.py", line 253, in restore_books
File "site-packages\calibre\db\cache.py", line 57, in call_func_with_lock
File "site-packages\calibre\db\cache.py", line 1985, in restore_book
File "site-packages\calibre\db\cache.py", line 1493, in create_book_entry
File "site-packages\calibre\db\backend.py", line 804, in execute
File "c:\cygwin64\home\kovid\sw\build\apsw-3.8.2-r1\src\cursor.c", line 231, in resetcursor
ConstraintError: ConstraintError: UNIQUE constraint failed: books.id

There is a long list of books with this same error.If I look for them, they are on the database and the majority opens just fine. While in other cases it tells me that I have "no available formats" Why?

Also, I now have also 6 books with "no available formats" AND with no "size" in the relevant column (makes them easy to spot), but opening the path shows that I actually have 2 PDFs, but anyway, I guess I can re-add them manually, it is only 6 after all.

What's going on? Should I do something differently? Would it be a good idea to reinstall again but this time delete all the previous metadata file from the Calibre library?

eschwartz · 08-20-2015, 02:50 PM

Error:

Code:

ConstraintError: ConstraintError: UNIQUE constraint failed: books.id

You had two folders with the same number at the end. That number is the database id, so calibre could only add one of them, and warned you that the other ("Jack Doe\Your Computer Sucks (271)") will not be added.

If you use calibre's Move library feature, it will leave behnd all the errored folders, and you can add them manually. Re-adding them will create a new id.

There is a good chance you will have duplicates. e.g. the book was locked in a folder, but the metadata changed and the OPF+cover was moved to a new location. Make sure to connect up the metadata with the ebook files.

BranMakMorn · 08-20-2015, 03:01 PM

Quote:

Originally Posted by eschwartz

Error:

Code:

ConstraintError: ConstraintError: UNIQUE constraint failed: books.id

You had two folders with the same number at the end. That number is the database id, so calibre could only add one of them, and warned you that the other ("Jack Doe\Your Computer Sucks (271)") will not be added.

If you use calibre's Move library feature, it will leave behnd all the errored folders, and you can add them manually. Re-adding them will create a new id.

There is a good chance you will have duplicates. e.g. the book was locked in a folder, but the metadata changed and the OPF+cover was moved to a new location. Make sure to connect up the metadata with the ebook files.

OK, doing it now. By "re-add" you mean take the PDF files out of the leftover folders and add them to Calibre, right? (so that it creates a new indexed folder for each of them). If I just copy and paste the whole folders into the new library nothing good will happen I suppose.

eschwartz · 08-20-2015, 04:24 PM

You got it.

BranMakMorn · 08-21-2015, 04:19 AM

Quote:

Originally Posted by eschwartz

You got it.

OK, very last question. Any guess of how it happened that about 10 books disappeared (the PDF) and got replaced by a "thumbs.db" file?

eschwartz · 08-21-2015, 01:56 PM

Thumbs.db is windows caching garbage -- see https://en.wikipedia.org/wiki/Windows_thumbnail_cache

The PDF files disappeared in an entirely unrelated manner.

LadyKate · 08-25-2015, 09:42 AM

I find it strange that you would trust your main library within the dropbox folder. I do hope you also have a backup of it elsewhere.

If it was my library, with all that work classifying things etc as well as being the main storage of my books, I would have my working library on a normal drive. If I wanted to have cloud storage, I would backup to that cloud storage folder.

What happens to files while storing within the cloud storage folder (dropbox, google etc) is not predictable. I consider my library to be too important to not have multiple backups. Yes, a rar backup is rather old school but it is one of the methods I use to get a "snapshot" of the library every few months. I also use FreeFileSync to do regular backups when I make changes. I keep these backups on different drives in case of drive failure.

For files that go to my tablet, I save them from Calibre in the file structure I wish to have on the tablet.

This works well for me and might not for others but I would urge everyone to start making backups. Use separate drives and/or a memory stick.

I guess the effort we tend to put into backups is relative to the importance of the data to us. I just thank goodness for cheap usb drives and memory sticks instead of needing to use those horrid tape backup drives.

08-20-2015, 02:50 PM	#21
eschwartz Ex-Helpdesk Junkie Posts: 19,421 Karma: 85400180 Join Date: Nov 2012 Location: The Beaten Path, USA, Roundworld, This Side of Infinity Device: Kindle Touch fw5.3.7 (Wifi only)	Error: Code: ConstraintError: ConstraintError: UNIQUE constraint failed: books.id You had two folders with the same number at the end. That number is the database id, so calibre could only add one of them, and warned you that the other ("Jack Doe\Your Computer Sucks (271)") will not be added. If you use calibre's Move library feature, it will leave behnd all the errored folders, and you can add them manually. Re-adding them will create a new id. There is a good chance you will have duplicates. e.g. the book was locked in a folder, but the metadata changed and the OPF+cover was moved to a new location. Make sure to connect up the metadata with the ebook files.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Big Library Read	Fbone	General Discussions	7	06-09-2015 06:02 PM
Big Library (accelerate starting?)	epubfan	Library Management	1	12-28-2013 02:58 PM
Sudden increase in problems	Darnielle	Kobo Tablets	29	12-10-2011 08:02 AM
Sudden big slowdown in page turn time	KlondikeGeoff	Sony Reader	16	06-25-2011 10:15 PM
New Owner - Big Problems	Crumbz	iRex	9	11-04-2008 12:40 AM

08-20-2015, 04:34 AM	#18
BranMakMorn Enthusiast Posts: 30 Karma: 10 Join Date: Jan 2010 Device: none	Well I did use the fix, but it fixed a very small amount of errors. I still find myself with several missing PDFs (luckily, I think only something like 20-30 rather than over 100). As well as a LOT of folders with two copies of the same PDF, or two copies of the metadata, or the cover file, which calibre sees in the database but tells me has no associated file. I am manually fixing the issue (deleting the extra files) for the few hundred cases I can easily identify (those titles Calibre tells me have no "size" in Mb). I've already closed Google Drive. Once I'm done with this I will uninstall, reinstall and put my library into OneDrive, hoping that in the process of reconstituting the library Calibre will "find" those files which now says are missing but are not. As for the lost books, that's kind of a mystery because it seems to me that the total size of the Calibre folder is the same.

08-20-2015, 09:00 AM	#19
PeterT Grand Sorcerer Posts: 13,535 Karma: 78910202 Join Date: Nov 2007 Location: Toronto Device: Libra H2O, Libra Colour	There is no need to uninstall / reinstall calibre to create a new library; just right click on the icon in the toolbar that shows the library, and select the option Switch/create library... and make your new library where you want it.

08-20-2015, 12:08 PM	#20
BranMakMorn Enthusiast Posts: 30 Karma: 10 Join Date: Jan 2010 Device: none	OK so: 1) I moved the Calibre Library out of Google Drive, and it is now in the OneDrive folder, but I paused the sync immediately, to make sure I could fix the library via Calibre first, and sync later; 2) I reinstalled Calibre, and told it to look for a previously existing library in the new directory; 3) I did a "restore database", which took ages, but gave me back several errors. I give an example here calibre, version 2.35.0 WARNING: Success: Restoring the database succeeded with some warnings. Click "Show details" to see the details. The old database was saved as: C:\Users\XXXXX\OneDrive\Documents\Calibre Library\metadata_pre_restore.db Failed to restore the books in the following folders: C:\Users\XXXXX\OneDrive\Documents\Calibre Library\Jack Doe\Your Computer Sucks (271) with error: Traceback (most recent call last): File "site-packages\calibre\db\restore.py", line 253, in restore_books File "site-packages\calibre\db\cache.py", line 57, in call_func_with_lock File "site-packages\calibre\db\cache.py", line 1985, in restore_book File "site-packages\calibre\db\cache.py", line 1493, in create_book_entry File "site-packages\calibre\db\backend.py", line 804, in execute File "c:\cygwin64\home\kovid\sw\build\apsw-3.8.2-r1\src\cursor.c", line 231, in resetcursor ConstraintError: ConstraintError: UNIQUE constraint failed: books.id There is a long list of books with this same error.If I look for them, they are on the database and the majority opens just fine. While in other cases it tells me that I have "no available formats" Why? Also, I now have also 6 books with "no available formats" AND with no "size" in the relevant column (makes them easy to spot), but opening the path shows that I actually have 2 PDFs, but anyway, I guess I can re-add them manually, it is only 6 after all. What's going on? Should I do something differently? Would it be a good idea to reinstall again but this time delete all the previous metadata file from the Calibre library?

08-20-2015, 04:24 PM	#23
eschwartz Ex-Helpdesk Junkie Posts: 19,421 Karma: 85400180 Join Date: Nov 2012 Location: The Beaten Path, USA, Roundworld, This Side of Infinity Device: Kindle Touch fw5.3.7 (Wifi only)	You got it.

08-21-2015, 01:56 PM	#25
eschwartz Ex-Helpdesk Junkie Posts: 19,421 Karma: 85400180 Join Date: Nov 2012 Location: The Beaten Path, USA, Roundworld, This Side of Infinity Device: Kindle Touch fw5.3.7 (Wifi only)	Thumbs.db is windows caching garbage -- see https://en.wikipedia.org/wiki/Windows_thumbnail_cache The PDF files disappeared in an entirely unrelated manner.

08-25-2015, 09:42 AM	#26
LadyKate Fanatic Posts: 515 Karma: 1470724 Join Date: Jul 2013 Location: Quebec CA Device: android 4 (samsung tablet and asus tablet)	I find it strange that you would trust your main library within the dropbox folder. I do hope you also have a backup of it elsewhere. If it was my library, with all that work classifying things etc as well as being the main storage of my books, I would have my working library on a normal drive. If I wanted to have cloud storage, I would backup to that cloud storage folder. What happens to files while storing within the cloud storage folder (dropbox, google etc) is not predictable. I consider my library to be too important to not have multiple backups. Yes, a rar backup is rather old school but it is one of the methods I use to get a "snapshot" of the library every few months. I also use FreeFileSync to do regular backups when I make changes. I keep these backups on different drives in case of drive failure. For files that go to my tablet, I save them from Calibre in the file structure I wish to have on the tablet. This works well for me and might not for others but I would urge everyone to start making backups. Use separate drives and/or a memory stick. I guess the effort we tend to put into backups is relative to the importance of the data to us. I just thank goodness for cheap usb drives and memory sticks instead of needing to use those horrid tape backup drives.