Calibre not working (db error?)

[manawydan]

Ok, I am completely stuck.

I have added a big amount of ebooks to Calibre (in 200-800 steps).
Suddenly I could no longer open Calibre.
Whenever I tried I got the error message (programme had to be closed, sorry for inconveniance etc.).

Well, ok. I downloaded the newest version (1.18) and installed (overwriting the existing programme).
Same result.

I downloaded the newest portable and installed on another harddisc. This works fine but it is not possible to switch the library to the old one.
This library is really big (approx. 112 GB+).

I thought that maybe it will work if I put the library on a disc that has only few data yet so I copied the library folder to the disc with the portable Calibre software and tried to switch library again.
Not possible.

There is nothing happening, Calibre is not responding (even after hours) and/or I get an error message.

What else can I do now?
I have a backup of the Calibre profile as well as library which is some weeks old - way before I tried to add the amount of ebooks.

Could it be a problem with the metadata_db?
The software itself seems to be ok otherwise it wouldn't work after installing and with just 1 file in its library.

Should I completely delete the original Calibre programme, install new and try to direct it the library folder (but I can't imagine that this will help)?

OS: WinXP, sp3
Pls. no "you must upgrade to Win7 or 8". It is not possible now. I know that XP is outdated but Calibre worked (more or less fine) with it until recently.

[theducks]

112GB is a pretty insane value for a normal book library

That said. As long as the files have room, documents (the formats) size should not matter.

How many Records (titles) so far, in your Library?

Have you checked the Library drive for disk errors?
Caution: don't do anything until Kovid responds... There might be a recovery that is faster than starting over.

[manawydan]

I think Calibre showed approx. 80k books but probably 10-20k are doubles which I intended to remove after everything had been added.

Yes, there is enough room on the hd. I have copied it to another hd as well and it is very unlikely that two discs have problems (both are quite new)

[BetterRed]

@manawydan - copy the the database (metadata.db) somewhere, then browse the author & book tables (in the copy) with something like SQLite Browser, if it barfs and produces some diagnostics then they may be useful in helping you recover, so squirrel them away somewhere.

If you have to rebuild from scratch then do it in batches, as I gather you did last time - but this time at the end of each batch backup your library. Seems like you have plenty of local disk space so I suggest you avoid cloud based solutions for now. If you use something like Freefilesynch (FFS) it only copies stuff that's changed, FFS has a good reputation for both function and performance.

Is the disk drive on which you have the library formatted for NTFS - I'm not sure why, but the thought of a 112GB library on a FAT formatted disk is a bit...

BR

[manawydan]

Originally Posted by BetterRed

@manawydan - copy the the database (metadata.db) somewhere, then browse the author & book tables (in the copy) with something like SQLite Browser, if it barfs and produces some diagnostics then they may be useful in helping you recover, so squirrel them away somewhere.

Ok, will try.

Is the disk drive on which you have the library formatted for NTFS - I'm not sure why, but the thought of a 112GB library on a FAT formatted disk is a bit...

Both drives are NTFS. In fact all my harddisc are NTFS.

P.S.
Just a thought - would it be of use to rename or move the db and copy the backup-ed db into the folder instead?
And somehow remove the files in the library that were added after the backup and therefor not in this backup version of the db?

I have asked different people if Calibre has a limit it can handle but everybody said no.
But either it has or there was a problem with the db or disc .. hm.

[BetterRed]

Originally Posted by manawydan

Both drives are NTFS. In fact all my harddisc are NTFS.

Well that's a blessing

Originally Posted by manawydan

Just a thought - would it be of use to rename or move the db and copy the backup-ed db into the folder instead?
And somehow remove the files in the library that were added after the backup and therefor not in this backup version of the db?

Yes maybe - but do it on a copy of library, make sure you keep the corrupt library intact :

Originally Posted by theducks

Caution: don't do anything until Kovid responds... There might be a recovery that is faster than starting over.

A Calibre library is the entire library folder - ie the author and book folders, the metadata.db and any other files you see there. You should always backup the library folder and everything in it - yes it takes a bit longer, but if you use something like FFS and you do it regularly its not too onerous.

So, copy that backup of metadata.db into the library copy (overwrite the one that's there) now try to start calibre, if it starts run the Library Maintenance->Check Library function - both phases. The second phase should tell you which books are in the library folder but not in the database.

You could move those book folders out of library and use them as the source to re-add the books

Or you could try running Library Maintenance->Restore database function - warning this may take a long time and it may not solve the problem.

Originally Posted by manawydan

I have asked different people if Calibre has a limit it can handle but everybody said no.
But either it has or there was a problem with the db or disc .. hm.

The limits are more likely to be imposed by the file system and the SQLite database, and I think you're a long way short of those.

When you add books to calibre then, depending on the Add Books config settings you have, calibre will read the format files - its always possible that one of the format files has led to the database corruption - I'm not aware that it's ever happened, but with 3.5m active users how would anyone know.

BR

[Adoby]

When you work with big amounts of files and data like this, it is a good idea to make backups now and then. In your case copy the whole library to some other hard drive. Then, if/when there are problems you only have to restore the last working backup, and don't have to start over from the beginning.

Also I would suggest that it is a bad idea to add everything to calibre, and create a huge library full of junk and duplicates. I don't think you'll ever sort it out. Even if calibre can handle all the books, everything will be slow and sorts and searches and updates and changes takes a looong time. And most likely you spend a lot of that time on books you will delete later.

Instead add just a few books, 20-50, perhaps books you want to read soon, or books that you know are in a good state. And fix them so they are in a good or even perfect state with nice covers, complete metadata, tags, normalized author names, correct series and converted to the format(s) you need and safely backuped. Then add a new batch. Preferably the best and most interesting books first. Rinse and repeat. I believe that this actually will be as fast, since I suspect that you'll never finish, and you'll at least have a nice growing library as you work, instead of an embarrassing ugly heap of junk with a few nice nuggets in it, here and there.

You may also use more than one library. For instance one for fiction and one for non-fiction.

[theducks]

Originally Posted by Adoby

You may also use more than one library. For instance one for fiction and one for non-fiction.

And they don't even need to be on the same spindle (physical drive).

Having a Logical split like Fiction and Not Fiction

Also allows for a different set of custom columns, an entirely different set of Tags (a smaller Tag DB), Smaller Author list...

[mbovenka]

Originally Posted by BetterRed

The limits are more likely to be imposed by the file system and the SQLite database, and I think you're a long way short of those

Yep. My library is roughly the same size as his (90K books, ca, 100GB) and I ran into an interesting issue when I was backing it up to my (FAT32-formatted) 128 GB USB stick. FreeFileSync bombed out with errors that seemed to point at a lack of space, although there was room enough on the stick.

Turned out it was a FAT32 LFN -> 8.3 Short Name mapping collision...

Converting the stick to NTFS fixed it.

[manawydan]

Originally Posted by Adoby

Also I would suggest that it is a bad idea to add everything to calibre, and create a huge library full of junk and duplicates.

Well yes and no. I see the point in your suggestion but my library was not that ugly.

I regularely did duplicate checks, metadata was sufficient organised (don't need tags or covers but authros, series, titles, formats and in most cases duplicates were checked and mantained).

And is it not the purpose of a database to collect all data and allow a quick search?
Real libraries do have the books stored in different places but there must be one central organising catalog (aka database) or you will be lost.

You may also use more than one library. For instance one for fiction and one for non-fiction.

I will certainly try a rough splitting. Actually I planned to do this for languages but as with so many projects it always was something to try after the holidays, during the next vaccations, when there is nothing else to do and so on ..

And it was quite conveniient to see at one glance german and english works by one author. If I now split it and want to get the same result I will have to switch library (since I do not think you can run two instances of Calibre simultanously) which - with still big libraries - means waiting time etc.
Not really tempting.

But well, lets see. First the whole thing must run again.

Originally Posted by mbovenka

Yep. My library is roughly the same size as his (90K books, ca, 100GB)

Thanks God, I am not alone.
He is a she btw.

Do you have any problems with your library? Very long loading times for example?

[mbovenka]

Originally Posted by manawydan

Thanks God, I am not alone.
He is a she btw.

Do you have any problems with your library? Very long loading times for example?

Sorry, default assumptions striking .

No, I don't have any issues with my library. Even on my old laptop (1,5GB Core2Duo WinXP) Calibre started in ~30 seconds, and on my new one (8GB Core i7 Win8.1) it's less than half that. Searching and editing was quite workable on the old one and is very quick now. I've always been very impressed with Calibre's performance, especially with the new database backend; that sped up things quite a bit.

[manawydan]

Originally Posted by mbovenka

No, I don't have any issues with my library. Even on my old laptop (1,5GB Core2Duo WinXP) Calibre started in ~30 seconds, and on my new one (8GB Core i7 Win8.1) it's less than half that. Searching and editing was quite workable on the old one and is very quick now. I've always been very impressed with Calibre's performance, especially with the new database backend; that sped up things quite a bit.

Curious.

I have had problems ever since the file count went up.
I thought the db was simply too big.
Then I suspected the old PC.
And then I got a brand new one and it was actually worse so I thought it is WinXP in combination with the hardware. And of course that the db is that big.

But it seems the problem lies somewhere else.

[theducks]

Originally Posted by manawydan

And it was quite conveniient to see at one glance german and english works by one author. If I now split it and want to get the same result I will have to switch library (since I do not think you can run two instances of Calibre simultanously) which - with still big libraries - means waiting time etc.

Single Language viewing is a perfect use of a Virtual Library as I would not exepct the Library structure would need to be different.

[BetterRed]

Originally Posted by manawydan

Curious.

I have had problems ever since the file count went up.
I thought the db was simply too big.
Then I suspected the old PC.
And then I got a brand new one and it was actually worse so I thought it is WinXP in combination with the hardware. And of course that the db is that big.

But it seems the problem lies somewhere else.

@manawydan - FWIW I have a 590GB library, ~200 authors, ~7600 'books', it contains lots of audio and video clips and some transcripts

The slow startups can be caused by having

• on pre 1.0 calibre versions showing formats as a column in the library list could lead to slow starts, but that was addressed in 1.0 (new db backend) see https://www.mobileread.com/forums/sho...ximate_formats
  
• custom columns built from other columns can slow calibre startup and database switch times. I had an example in that 590G database, when I reached ~6000 books startup times went from a few seconds to several minutes. One of my columns was built from a union of three name columns - when I removed the column the startup times went back to a few seconds.

Have you been able to recover/restore your library ?

BR

[manawydan]

Originally Posted by theducks

Single Language viewing is a perfect use of a Virtual Library as I would not exepct the Library structure would need to be different.

But this would still mean all files are in one directory and there is just one db?
It is just a matter of displaying.
How can this help handling a big db?

Originally Posted by BetterRed

The slow startups can be caused by having
on pre 1.0 calibre versions showing formats as a column in the library list could lead to slow starts,

Well, it is 1.18 now.
I can of course try to not show the formats column.

custom columns built from other columns can slow calibre startup and database switch times. I had an example in that 590G database, when I reached ~6000 books startup times went from a few seconds to several minutes.

Yes I know and this is not the case.
The only customn columns I have are to show whether or not a book has been read or is on the e-reader and these columns are not based on others.

But back to the more pressing issue.

The good news is that after I pasted the old db over the current Calibre started (as usual very slow but at least it worked).
The bad news is that I obviously can not run a maintenance. Calibre crashed.

I think the next step should be to reduce the library and db by removing a big part (e.g. german language files).
Then try maintenance again and if this works rebuild what has been lost in the db by adding these files again (or however the best way would be to do this).

What do you think?

Not quite sure how to do it though - Calibre can display all e.g. German language files and I can mark them. And? Create Library and Copy these books to the newly created one?