Calibre not working (db error?)

[BetterRed]

@manawydan - re startup times - have you tried doing a defrag on the drive where the library is located ?

Originally Posted by manawydan

I think the next step should be to reduce the library and db by removing a big part (e.g. german language files).
Then try maintenance again and if this works rebuild what has been lost in the db by adding these files again (or however the best way would be to do this).

What do you think?

Not quite sure how to do it though - Calibre can display all e.g. German language files and I can mark them. And? Create Library and Copy these books to the newly created one?

How about

1. create a new Library based on existing one
2. use the Copy to library function to move books in batches - I would start with 200, and increase to 500 after first 1000.
3. after each batch check run Library Maintenance->Check library function - it should always get a clean bill of health - I think
4. do a library backup as previously suggested after each batch

I don't even like having separate 'books' for different languages, let alone separate libraries. So I coerce all versions of the same work (languages, editions, medium etc) into the same 'book'

BR

[theducks]

Originally Posted by manawydan

Not quite sure how to do it though - Calibre can display all e.g. German language files and I can mark them. And? Create Library and Copy these books to the newly created one?

Yes (after preparing a New-similar structured Library using that option on Create Switch)

search: languages:"=English"
Select All (results)
right-click Copy to Library:<you made above>: the choice is yours, but I wold NOT delete until I was positive of a Solid new Library

[theducks]

@br
I defrag, but I am not sure how that significantly helps speed in this case.
The only file that might benefit (startup) is metadata.db.

I believe the OP mentioned this is a USB drive. That will slow things down unless it can fully use a USB3 connection)

Make sure that Windows can NOT use this drive for swap (VM),
Exclude this drive from the Indexing Service
Exclude this drives Library folder from Realtime scans, all the files should have been pre-qualified in their original location

[BetterRed]

Originally Posted by theducks

@br
I defrag, but I am not sure how that significantly helps speed in this case.
The only file that might benefit (startup) is metadata.db.

Yeah, probably won't make much if any difference, makes a difference on XP for things like the zillion selfies that some people like to have, but they probably don't use XP

Originally Posted by theducks

I believe the OP mentioned this is a USB drive. That will slow things down unless it can fully use a USB3 connection)

Don't think manawydan did - mbovenka did though

My USB3 WD Caviar Blacks go faster than my internal Sata 2 ones

BR

[theducks]

Originally Posted by BetterRed

My USB3 WD Caviar Blacks go faster than my internal Sata 2 ones

BR

:Jealous: My newest (laptop) still only does USB2 (it does have eSATA port, but nothing else does)

[BetterRed]

I used eSata before I added the USB3 controller and disk dock - it only seemed to be about 10% faster than USB2.

I guess you can't add USB3 to a laptop. I'm thinking my next workstation build won't have any spinning drives, just one of these and a 2 slot USB 3 disk dock. USB3 controllers and docks don't cost too much, but the WD caviar blacks ain't real cheap.

[theducks]

Originally Posted by BetterRed

I used eSata before I added the USB3 controller and disk dock - it only seemed to be about 10% faster than USB2.

I guess you can't add USB3 to a laptop. I'm thinking my next workstation build won't have any spinning drives, just one of these and a 2 slot USB 3 disk dock. USB3 controllers and docks don't cost too much, but the WD caviar blacks ain't real cheap.

They are lots cheaper than my first 40Meg (MFM) Seagate

[manawydan]

Originally Posted by theducks

I believe the OP mentioned this is a USB drive. That will slow things down unless it can fully use a USB3 connection)

No, I only used Calibre Portable because I didn't want to worry about interfering with the original Calibre profile on disc C.

The orignal (crashing) programme is on disc C. It's library on disc D.

The new portable install is on disc G and this versions library is on G as well.

There is one thing that you very likely do not have on your systems but which I very much need though I do not know if and why it would interfere - it's a screen magnifyer called Zoomtext. This is also the reason why I use WinXP (apart from that I really do not like the handling of Win7).
It is a little bit complicated so lets just say for the moment WinXP is a must.

I can not test the system without it though since running in secure mode or normal mode but with ZT deinstalled (simply deactivating does not change anything) I do not see anything on screen. *sigh*

However there is no other software that acts like Calibre/the database so I very much doubt ZT is the reason.
And if the library is very small Calibre works fine with ZT on. So, no, I really don't think this is the cause.

I will try to get this to work now, thanks for your recommendations.

[BetterRed]

Originally Posted by manawydan

And if the library is very small Calibre works fine with ZT on. So, no, I really don't think this is the cause.

Agreed and for XP+NTFS

BR

[manawydan]

First, the good news is, the new portable Calibre starts up in about 1 minute (compared with up to 20+ minutes before ). The DB is smaller but not that much smaller. Funny.
Another indicator poiting to Windows and the harddisc I think.

Bad news is that a library switch or a maintenance of the big DB still crashes Calibre.

However when it starts up again it does with the last selected library and it does it in about 1 minute or less.
Maintenance of a newly created, small library works fine so far and shows no errors.

Ok, I then thought I restore the original settings from the backup step by step. Just to see what will happen.

I restored gui, global, customize (which did not exist in portable), shortcut and tweaks.
I noticed that although I did not restored the plugins folder the user installed plugins are there.
If I rename the old calibre settings folder in the applications data on C the plugins on G don't work.
So one of the restored files controls plugins and connects to harddisc C.

So, how to best solve it?

Allow Calibre Portable on G to connect to the pluginfolder on C and then deinstall/disable all user plugins and install again?
Some can only be disabled but not removed (un-DRM for example).

Or go back a step to the original settings of Calibre Portable and do all changes and tweaks etc. manually again (hope not to have to since there are several ones)?

Which file controlls the use of plugins? global? customize?

[BetterRed]

Originally Posted by manawydan

First, the good news is, the new portable Calibre starts up in about 1 minute (compared with up to 20+ minutes before ). The DB is smaller but not that much smaller. Funny.
Another indicator poiting to Windows and the harddisc I think.

Why do you think that.

The database (metadata.db) is never very big, I doubt there are many bigger than a 100MB, the biggest I have 18MB for a 30,000+ book library.

To show Windows is not causing your problems: can you take your G: drive, containing calibre portable and your bad library, to a friend or relative and use their computer to do a test. I appreciate the other computer won't have Zoomtext, but perhaps they can help you, or if they have a laptop they could bring it to your place. If you get the same problem on another computer then that would more than convince me that Windows is not the cause of your problem.

As I understand your posts you've done tests of the same library on three drives - C, D and G - and gotten the same error. So, IMO, its most unlikely to be a drive problem. But if you were able to copy the library to a friends computer and do a test and it fails, then that should put the bad disk theory to bed.

Originally Posted by manawydan

Bad news is that a library switch or a maintenance of the big DB still crashes Calibre.

Which IMO is further evidence that the problem is almost certainly in the library.

You can copy most of your installed calibre configuration folder to the ...Calibre Portable\Calibre Settings folder, but I don't know which files and folders are not OK to copy.

Based on everything you've tried (different drives, different computers, different versions of calibre) the symptoms are that the library - probably the metadata.db component - is broken. So recreating your installed calibre configuration in calibre portable is unlikely to change that - my prediction is that it will still crash.

IMO you're going to have to rebuild the library, suggestions have been made on how you could do that. And again IMO, everything else - Windows XP, bad disk drives, Calibre configuration data, Zoomtext, languages etc - are diversions.

BR

[eschwartz]

Originally Posted by BetterRed

You can copy most of your installed calibre configuration folder to the ...Calibre Portable\Calibre Settings folder, but I don't know which files and folders are not OK to copy.

I believe you can just copy over everything. You should be fine, except for the saved library paths, but you would have to reset those anyway since the library is moved.

It is exactly the same code to create and utilize both settings folders, but one copy comes bundled with a script to redirect calibre when it's looking for the settings folder location.

Now, if you tried to copy over from a linux settings folder, we'd have a problem...

[BetterRed]

Before I made my most recent post in this thread I copied the contents of my calibre 64 bit installed config folder to a fresh Calibre Portable\Calibre Settings folder.

Then I started calibre portable (by hitting enter on Calibre Portable\calibre-portable.exe in Windows explorer). According to calibres status line, and Windows Task Manager and Process Explorer I was running calibre 64 bit.

I restored the Calibre Portable\Calibre Settings folder back to its install state.

Then I started calibre portable (by hitting enter on Calibre Portable\calibre-portable.exe in Windows explorer). According to calibres status line, and Windows Task Manager and Process Explorer I was running 32 bit calibre via calibre-portable.exe

From that I concluded you cannot copy the entire contents of a 'regular' configuration folder to Calibre Portable/Calibre Settings. I suspect you can copy most of it, but I am unwilling to conjecture on what you can and cannot copy on an item by item basis.

BR

[eschwartz]

I can see there being a difference in the 32-bit build vs. the 64-bit build but the 32-bit build IS the calibrePortable build just with a scripted startup to point it at a different location for the config folder and default library.

Although it is a good point and should be taken into consideration that moving the config over to a new installation may be dependent on using the same-bit version as the old one.

[BetterRed]

@eschwartz - I have both the 32bit and 64bit versions of calibre installed, AFAIK by default they use the same configuration data. Not a copy thereof - the same data, on the same drive, in the same folders using the same disk sectors. If I change a preference from the 32bit version, close it and start the 64bit version I see the change I just made to preferences from the 64bit version.

To make the 32bit & 64bit installs use different config data I think I would have to poke different values into the CALIBRE_CONFIG_DIRECTORY environment variable before running them - I've done that too, so I know it works as I expect to work.

BR