OneDrive problem (?)-Calibre Library not including all books in library

gandoe · 11-10-2024, 12:14 PM

So, I think I've created my own problem. I had been running Calibre on my desktop system w/no problem; library files (and anything else in the MyDocs folder tree) mirrored monthly to a NAS. My Calibre backups are more sporadic. About 3 months ago, I moved the entire MyDocs folder tree into Microsoft OneDrive; still local on my machine, but also accessible via the web.

About a month ago, I recently purchased a new laptop; and used the same Microsoft credentials to set up, so that the Microsoft OneDrive would then also be sync'd to the laptop.

Well, somehow my versions of the Calibre library got unsync'd because I was in the process of adding some new books, and accidentally added an older book which I knew was already in my library; but never received the relevant Calibre message about combining records.

Short story, it appears that all my books are in the Calibre Library folder tree, but a fair number are not in my Calibre database. Running a Check Library shows some 150 "Extra titles", files in the library without corresponding data base entries.

It is possible that I may have committed a classic "Doh" error...I use both an Onyx Boox eInk and Samsung Tab8 Pro for reading, and in syncing, adding and removing books from either or both devices, I may have accidently "Remove Selected Books" rather that "Remove Matching Files from Device"

I assume I could restore an older backup file, and then simply add back the newer additions; however, my actual Calibre Backups are not very recent; I didn't recognize that the mirrors of the Calibre Library weren't the actual Calibre database files, located in the program files directory. Is there any file locations which might have prior copies of the Calibre database?

Otherwise, does anyone have any suggestions about either somehow restoring a prior copy of the database files (if I can find them) or otherwise adding back all those "Extra titles"?

TIA
Ed

chaley · 11-10-2024, 12:51 PM

First: don't put your library on OneDrive or Dropbox or a NAS or an SMB mount or anywhere else a network or syncing is involved. See https://manual.calibre-ebook.com/faq...rked-drive-nas. I wish calibre could detect this situation because we get multiple "Help, my library has worked for (some long time) but is now corrupt!" posts per week caused by this issue.

My guess is that unless you have a recent backup of the full library (one of your mirrors might work but they easily could be equally corrupt), you are in for some work. That said, some people have had luck with this process:

Make a full backup of the existing library! Actually copy the library folder to a local drive.
Start calibre. It should open the old library.
Create a new library in a folder on a local unsynced file system using "Choose Library / Create an empty library at the new location", with "Copy structure from the current library" checked. This will create and open an empty library.
Switch back to the old library.
Use "Copy to library, delete after copy" to move all books to the new library. This will leave behind all the books that have been "lost".
Close the old library.
Open the new library.
Do "Add from folders and subfolders." Answer Yes to the question "Assume all e-book files in a single folder are multiple formats of the same book?" Choose the old library folder as the source.
If you are lucky your library will be mostly correct. The metadata might need some help as changes since the damage will be lost, and you should check item notes (author, series, etc) if you use them.

gandoe · 11-10-2024, 01:09 PM

Got it...thanks for the help. I knew not to install Calibre on a network drive, but somehow didn't make the connection that OneDrive is similar to a network drive....doh! Yet another unforced error!

I'll have to add back the metadata...oh well.

Thanks!

gandoe · 11-10-2024, 01:48 PM

Ah...I just discovered a copy of the metadata data base file that was left behind in the original Calibre Library folder before I moved it to the OneDrive folder. Guess it didn't want to be moved. Is there some way to use this file to restore the original metadata, and then simply add new metadata for books added after that date?

chaley · 11-10-2024, 02:45 PM

All the metadata is in the db. You can try copying the file into your library and see what check library says. Be sure to keep a copy of the original! If it is better then keep it, otherwise put the original back.

gandoe · 11-10-2024, 02:49 PM

Got it, thanks!

gandoe · 11-11-2024, 09:17 AM

OK, using the original metadata file, and following the instructions provided seemed to work. There were 367 books (out of 2,294 total books) that were missing from the Calibre database. I'll have to go through and add back my custom metadata...at least it appears my Calibre library is functional again. Whew! Thanks for your help!

11-10-2024, 12:14 PM	#1
gandoe Connoisseur Posts: 76 Karma: 9424 Join Date: Dec 2011 Device: none	OneDrive problem (?)-Calibre Library not including all books in library So, I think I've created my own problem. I had been running Calibre on my desktop system w/no problem; library files (and anything else in the MyDocs folder tree) mirrored monthly to a NAS. My Calibre backups are more sporadic. About 3 months ago, I moved the entire MyDocs folder tree into Microsoft OneDrive; still local on my machine, but also accessible via the web. About a month ago, I recently purchased a new laptop; and used the same Microsoft credentials to set up, so that the Microsoft OneDrive would then also be sync'd to the laptop. Well, somehow my versions of the Calibre library got unsync'd because I was in the process of adding some new books, and accidentally added an older book which I knew was already in my library; but never received the relevant Calibre message about combining records. Short story, it appears that all my books are in the Calibre Library folder tree, but a fair number are not in my Calibre database. Running a Check Library shows some 150 "Extra titles", files in the library without corresponding data base entries. It is possible that I may have committed a classic "Doh" error...I use both an Onyx Boox eInk and Samsung Tab8 Pro for reading, and in syncing, adding and removing books from either or both devices, I may have accidently "Remove Selected Books" rather that "Remove Matching Files from Device" I assume I could restore an older backup file, and then simply add back the newer additions; however, my actual Calibre Backups are not very recent; I didn't recognize that the mirrors of the Calibre Library weren't the actual Calibre database files, located in the program files directory. Is there any file locations which might have prior copies of the Calibre database? Otherwise, does anyone have any suggestions about either somehow restoring a prior copy of the database files (if I can find them) or otherwise adding back all those "Extra titles"? TIA Ed

11-10-2024, 12:51 PM	#2
chaley Grand Sorcerer Posts: 12,447 Karma: 8012886 Join Date: Jan 2010 Location: Notts, England Device: Kobo Libra 2	First: don't put your library on OneDrive or Dropbox or a NAS or an SMB mount or anywhere else a network or syncing is involved. See https://manual.calibre-ebook.com/faq...rked-drive-nas. I wish calibre could detect this situation because we get multiple "Help, my library has worked for (some long time) but is now corrupt!" posts per week caused by this issue. My guess is that unless you have a recent backup of the full library (one of your mirrors might work but they easily could be equally corrupt), you are in for some work. That said, some people have had luck with this process: Make a full backup of the existing library! Actually copy the library folder to a local drive. Start calibre. It should open the old library. Create a new library in a folder on a local unsynced file system using "Choose Library / Create an empty library at the new location", with "Copy structure from the current library" checked. This will create and open an empty library. Switch back to the old library. Use "Copy to library, delete after copy" to move all books to the new library. This will leave behind all the books that have been "lost". Close the old library. Open the new library. Do "Add from folders and subfolders." Answer Yes to the question "Assume all e-book files in a single folder are multiple formats of the same book?" Choose the old library folder as the source. If you are lucky your library will be mostly correct. The metadata might need some help as changes since the damage will be lost, and you should check item notes (author, series, etc) if you use them.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Reinstalling Calibre library after losing all the books in the library by accident.	Heathside Boy	Library Management	33	10-24-2022 05:48 AM
Problem moving books from Kindle into Calibre Library	peterglass	Library Management	2	05-30-2022 05:05 PM
Calibre issues - Missing libraries and problem loading books from new library	markysparks	Calibre	1	01-29-2020 07:42 AM
cannot select Calibre Library folder in Onedrive	dwl99	Calibre Companion	6	10-12-2016 02:55 PM
synchronize library Calibre in OneDrive	beatriz	Devices	3	03-05-2015 12:20 PM

11-10-2024, 01:09 PM	#3
gandoe Connoisseur Posts: 76 Karma: 9424 Join Date: Dec 2011 Device: none	Got it...thanks for the help. I knew not to install Calibre on a network drive, but somehow didn't make the connection that OneDrive is similar to a network drive....doh! Yet another unforced error! I'll have to add back the metadata...oh well. Thanks!

11-10-2024, 01:48 PM	#4
gandoe Connoisseur Posts: 76 Karma: 9424 Join Date: Dec 2011 Device: none	Ah...I just discovered a copy of the metadata data base file that was left behind in the original Calibre Library folder before I moved it to the OneDrive folder. Guess it didn't want to be moved. Is there some way to use this file to restore the original metadata, and then simply add new metadata for books added after that date?

11-10-2024, 02:45 PM	#5
chaley Grand Sorcerer Posts: 12,447 Karma: 8012886 Join Date: Jan 2010 Location: Notts, England Device: Kobo Libra 2	All the metadata is in the db. You can try copying the file into your library and see what check library says. Be sure to keep a copy of the original! If it is better then keep it, otherwise put the original back.

11-10-2024, 02:49 PM	#6
gandoe Connoisseur Posts: 76 Karma: 9424 Join Date: Dec 2011 Device: none	Got it, thanks!

11-11-2024, 09:17 AM	#7
gandoe Connoisseur Posts: 76 Karma: 9424 Join Date: Dec 2011 Device: none	OK, using the original metadata file, and following the instructions provided seemed to work. There were 367 books (out of 2,294 total books) that were missing from the Calibre database. I'll have to go through and add back my custom metadata...at least it appears my Calibre library is functional again. Whew! Thanks for your help!

Advert

Advert