Solution for this. Metadata.db gets corrupted. Waste more than 3 days fixing it.

[ygreq]

Hi folks!

I have been using Calibre since forever. And I always had this issue which is weird that it has not been solved till now.

I have a Calibre book library of about 60 000 books. And from time to time, the metadata.db gets corrupted. Usually this happens when I use the library to write something in it (like adding a new book or editing some book metadata or book title).

And whenever I try to do Library Maintenance -> Restore database, it takes like forever. Seriously it takes more than 3 days full 72 hours) or maybe more.

So considering that considering what I said about the probable reason why this database file gets corrupted is because of what I last did before actually corrupting itself, I was thinking that a backup copy of the metadata.db not older than a few minutes would definitely help in solving the issue a lot easier.

I would just:
-close Calibre
-delete the old corrupted metadata.db file,
-rename the backup file
-restart Calibre
-readd the book I was trying to add, or modify again the book details I was trying to modify. (All this would be based on "Modified" column sorting)

So any idea if this can be done? To have Calibre do a backup of metadata.db every 5 minutes? Or set the time in the Settings on how frequently you would want to do the backup.

So what do you say? Does it make sense?
Let me know!

PS: I was looking for 3rd party apps to backup files, but all the apps I found can backup a whole folder. So if you know of an app like that, it would be way around. At least until this hopefully gets solved. <3

[ownedbycats]

I take a daily backup of metadata.db, though more as a protection against user error. I'd suggest SyncBackFree -- it's got good filtering options that let you grab only the files you want.

But I'd look into the root causes of the corruption. Are you running on a local drive, or an NAS? Are you running it in something like Dropbox or GDrive? What's your antivirus?

[ygreq]

Hi, ownedbycats! Thank you for the recommendation. I checked the app, but it seems it also only backs folders and not specific files. I will look further. Maybe I am missing something.

Also I forgot to mention that I have multiple Calibre libraries, one for books, one for comicbooks and some others. I store them on external hard drives. I intend to run them in a NAS in the future.

The thing is that it does not happen often. And for sure I don't blame Calibre for corrupting the database file. It generally happens if there is some power failure or something else that happens while I use the app.

Now related to what I was saying earlier, an implementation within Calibre would backup the database file only while using that specific library. So that would be a lot nicer.

[ownedbycats]

Don't plan on running them on an NAS.

Originally Posted by ygreq

Hi, ownedbycats! Thank you for the recommendation. I checked the app, but it seems it also only backs folders and not specific files. I will look further. Maybe I am missing something.

You need to use the filtering option. "Choose sub-directories and files."

[theducks]

Calibre CAN rebuild the DB fairly quickly if nothing else was corrupted .

Every folder contains a Metadata.opf. That is the metadata that was in the Library at the time each entry was last updated.


You will see an entry: Metadata backup status
That is how many entries are still in the queue (usually the queue empties fast, except when you do something that affect a large number of books).
That is what is used If you right click the Library icon:Library Maintenance: Restore database.

This ASSUMES the rest of the Library files are intact (This is why a NAS/networked drive is dangerous and not to be used. Delays can cause pieces to be out of place/lost. )

Each Library (DB) is ONLY associated with the folders in the same Tree branch. Corrupt 'books' and your 'Comics' library was not involved.

THIS does not apply to a Hardware / Malware caused fail.

[PeterT]

When you mention external drives; how are they connected (USB 2, 3 or ....)

Sent from my Pixel 4a using Tapatalk

[BetterRed]

perhaps a Library Closed plugin could copy the metadata.db file of the library just closed to a metadata.db.backup fiie within the library folder, doing a replace of it already exists.

I say 'perhaps' because I don't know if LC plugins can have an event that kicks in when the metadata.db file is actually closed. The existing LC plugins operate on the content of the database (Virtual Library and Custom Column definitions) before its closed.

Recovery would be manual: Close calibre and copy metadata.db.backup to metadata.db, then run the calibredb check_library command and fix errors, OR start calibre, run Library Maintenance->Check library and fix errors.

BR

[ygreq]

Originally Posted by ownedbycats

Don't plan on running them on an NAS.



You need to use the filtering option. "Choose sub-directories and files."

Ah! I see. I will check asap. Thank you so much!

[ygreq]

Originally Posted by theducks

Calibre CAN rebuild the DB fairly quickly if nothing else was corrupted .

Every folder contains a Metadata.opf. That is the metadata that was in the Library at the time each entry was last updated.
.

Thank you . I know all of this. The problem is that once the medata.db file gets corrupted, it takes more than 3 days to recreate a new metadata.db file form the metadata.opf files for a 60 000 books library.

And this is the problem I was addressing.

[ygreq]

Originally Posted by PeterT

When you mention external drives; how are they connected (USB 2, 3 or ....)

Sent from my Pixel 4a using Tapatalk

They are connected via USB 2 and 3. Some older ones on 2 and newer on 3.

[ygreq]

Originally Posted by BetterRed

perhaps a Library Closed plugin could copy the metadata.db file of the library just closed to a metadata.db.backup fiie within the library folder, doing a replace of it already exists.

I say 'perhaps' because I don't know if LC plugins can have an event that kicks in when the metadata.db file is actually closed. The existing LC plugins operate on the content of the database (Virtual Library and Custom Column definitions) before its closed.

Recovery would be manual: Close calibre and copy metadata.db.backup to metadata.db, then run the calibredb check_library command and fix errors, OR start calibre, run Library Maintenance->Check library and fix errors.

BR

Interesting approach. But what if the fail happens while you are using calibre. For example the only fails I got was while using calibre. So I would still rather go with a metadata.db backup every n minute. And keep about 3 backups. And if shit happens, you just delete the old metadata.db file and rename the last backup. And see the folder structure of the entir elibrary and just re-add the last books you were trying to add or see which books were last modified to redo them.

[dunhill]

Another application to try is the one I use, Free File Sync

[ygreq]

I actually use FreeFileSync myself and loving it. And I actually use that to sync the whole calibre libraries. But no idea you could sync a single file. Can you do that?

I know RealTimeSync comes in the same package.

[ygreq]

@ownedbycats Ok. I don't get it. There seems to be lots of options, but I cannot find the filter you talk about. Can I ask for some easy steps? Or where exactly do I find that option?

Thank you so much!!

[ownedbycats]

Originally Posted by ygreq

@ownedbycats Ok. I don't get it. There seems to be lots of options, but I cannot find the filter you talk about. Can I ask for some easy steps? Or where exactly do I find that option?

Thank you so much!!

First, you create a new profile - Backup-type. I also suggest using the single-compressed file option as seen here:



Then it'll open up the options window. Put in your library folder in Source, and it'll enable the 'choose sub-directory and files' button.



Also, adding '%DATE%.zip' to the end of your destination will have it date your backups.