How to best start db from scratch?

manawydan · 04-20-2014, 06:37 PM

After my old db (or Calibre, I don't know) acts more and more funny I have decided to start from scratch.
On a laptop, Windows 7 OS, wit the newest Calibre version.

My question is how to best do it?
What would you recommend?

If I simply copy the old library folder - at once or step by step - over but not the .db and .json file (these are provided by the fresh installed Calibre) and then point Calibre to the folder with the copied library would this prevent problems that could mess up the database?

It is a big library with 100K+ database etnries.

eschwartz · 04-20-2014, 07:25 PM

calibre has a built-in utility to "Rebuild Database" from scratch. It is under Library icon ==> Libray Maintenance.

It will replace the db with a new one built from the metadata.opf backups, plus the .json with your settings backup.

BetterRed · 04-20-2014, 07:37 PM

Quote:

Originally Posted by manawydan

After my old db (or Calibre, I don't know) acts more and more funny I have decided to start from scratch.
On a laptop, Windows 7 OS, wit the newest Calibre version.

My question is how to best do it?
What would you recommend?

If I simply copy the old library folder - at once or step by step - over but not the .db and .json file (these are provided by the fresh installed Calibre) and then point Calibre to the folder with the copied library would this prevent problems that could mess up the database?

It is a big library with 100K+ database etnries.

@manawydan - I think you would need to rebuild the database via Library Maintenance->Library restore - that could take a quite a long time (overnight) if "100K+ database entries" means 100,000 'books'. The database will be rebuilt from the .opf files in the book files. If the database is the source of your problems, then rebuilding it from the .opf could replicate the problems.

Have you tried rebuilding the database on the current library, if you have and it didn't fix the problems, then I doubt that copying the author and book folders elsewhere and doing the same thing will resolve your issues - unless you have hardware problems.

It might be better to say what you mean by "my old db... acts more and more funny" - it could be the configuration data that's gone skew-whiff, rebuilding the database wont fix those problems.

Try renaming the config folder (with calibre not running) and then start calibre (it will build a fresh config folder) and point it to the existing library. You'll not have most customisations but it should be 'sufficient' to give you an indication of whether it is the library database that's causing your problems.

You can restore the old config by deleting the 'fresh' one and re-renaming the old one.

BR

manawydan · 04-25-2014, 04:01 PM

Quote:

Originally Posted by BetterRed

Have you tried rebuilding the database on the current library, if you have and it didn't fix the problems,

That is not possible.
Calibre can not handle any form of maintenance. It freezes and/or crashes.

To make a long story short, it is possible that Calibre is fine and it is a hardware problem or a hardware plus OS problem or a problem with a screen magnifyer or all together or nothing ..
I can not test it without the magnifyer anyway.

On the laptop there is Windows 7 and a 64bit newest version of this software.

Calibre is not the only software that has problems but with Calibre the problems are the obvious.

It does not have to be the .db - the newest trick are memory errors (though the only changes from when there were none to now is the amount of entries, sometimes the requested operation (e.g. sort, search etc.) will be executed anyway and sometimes I have to completely close the programme and start again.

However, this is a long and complicated story.

My hope now is that if I start from scratch on a new system I can avoid problems or at least see when they start to show up.

Quote:

Try renaming the config folder (with calibre not running) and then start calibre (it will build a fresh config folder)

You mean the setting folder (running Calibre Portable)?
And I leave the library as it is?
I can try but I am quite sure it will not work and Calibre will crash again.

But I'll give it a try, thanks.

eschwartz · 04-25-2014, 04:12 PM

Sounds like hardware/system problems, all right. If other programs are crashing too, that is almost a guarantee.
The db definitely is not responsible for those kinds of problems. Library Maintenance ==> Library Restore doesn't use the database, AFAIK. That's the whole point of it.

BetterRed · 04-25-2014, 06:30 PM

Quote:

Originally Posted by manawydan

You mean the setting folder (running Calibre Portable)?
And I leave the library as it is?
I can try but I am quite sure it will not work and Calibre will crash again.

Yes that's what a I meant - but I tend to agree with eschwartz, if other programs are misbehaving and you're getting memory faults then you probably have a hardware problem.

There's some discussion in this thread that is tangentially germane https://www.mobileread.com/forums/sho...light=graphics - in particular see chaley's comments towards the end of the thread where he discusses why different software may 'exercise' computer components in different ways etc.

I would run some memory tests. Also on laptops there is the problem of heat, the airflow ducts can become clogged quite easily, memory chips (main and video) are very susceptible to heat damage.

BR

theducks · 04-25-2014, 07:14 PM

+3

You have a basic, @ SYSTEM level failure .

Hardware dying, includes poor cooling or power rails failing.
You have a Malware infestation
A wrong driver was installed during an update..

(Admin level logon required) You should look at all RED items for clues, Review yellow items as some might be a warning of impending Hardware demise (I caught a HD going bad that way, while it was good enough to clone and go

)

Look at the Event viewer. All Programs: Administrative Tools: Event viewer

(Windows) System Log:

manawydan · 05-02-2014, 10:38 AM

Quote:

Originally Posted by BetterRed

Yes that's what a I meant - but I tend to agree with eschwartz, if other programs are misbehaving and you're getting memory faults then you probably have a hardware problem.

Yes unfortunately.

The PC is not that old though.
It is just not easy right now to find out thre reason.

Quote:

Also on laptops there is the problem of heat, the airflow ducts can become clogged quite easily, memory chips (main and video) are very susceptible to heat damage.

Yes I am aware of the heat problem.
I want to put it on the laptop as alternative until the PC problem is solved (if ever).
I can still use the PC-located Calibre but it is a pain in the ****.

The problems I now have are not on the laptop but the PC.

Cooling can't be a reason because I took good care to not only use better vents than came with the original frame but also get enough built in. Not that I know much about this but I got some advice from people who know about this.

Well of course there can always something go wrong.
And there can be plenty of other possible reasons.

Anyway, I will check out the link and thanks for your help.

BetterRed · 05-02-2014, 07:05 PM

@manawydan - I wonder of the magnifier puts extra load on the graphics adapter. As you will read in that thread I previously referenced I had some 'weird problems', not with calibre specifically, but generally. The cause was a flaky graphics adapter, but determining that it was the cause was in the end - a hunch. And a couple of friends who have had inexplicable problems solved them by replacing the graphics adapter.

BR

Dopedangel · 05-02-2014, 09:13 PM

You can copy your library to the new pc. If you are still getting problems use rebuild database on the new PC. I would recommend breaking up your library into smaller libraries generally something not recommended but with such a large library as your you would be better off.

BTW if you are using custom columns generated from other columns it will hit your calibre performance quite a bit as you have over 100K entries and they are generated on the fly

manawydan · 05-12-2014, 06:28 AM

Haven't made much progress (no time for dealing with Calibre) but wanted to thank you for the link.

The more I read the more I tend to agree that the magnifyer (Zoomtext) + the graphics system are good suspects.

I am now really curious how this will work on a Windows 7 64bit system with the newest ZT version.

I also noticed that recently there seem to be a black out (as mentioned in the linked posts - no blue scren just a immediate shutdown) when using a backup programme and trying to download a bigger file from a certain website (and only this site). However I managed to do both with ZT switched off and suddenly there were no black outs at all.

I am not sure what exactly this means but it's ... interesting.

No idea what a graphic adapter actually is or how to change it but I know someone who knows quite a bit about computers and hopefully knows what to do with the mentioned possible causes of the problem.

BetterRed · 05-12-2014, 07:55 AM

Quote:

Originally Posted by manawydan

Haven't made much progress (no time for dealing with Calibre) but wanted to thank you for the link.

The more I read the more I tend to agree that the magnifyer (Zoomtext) + the graphics system are good suspects.

I am now really curious how this will work on a Windows 7 64bit system with the newest ZT version.

I also noticed that recently there seem to be a black out (as mentioned in the linked posts - no blue scren just a immediate shutdown) when using a backup programme and trying to download a bigger file from a certain website (and only this site). However I managed to do both with ZT switched off and suddenly there were no black outs at all.

I am not sure what exactly this means but it's ... interesting.

No idea what a graphic adapter actually is or how to change it but I know someone who knows quite a bit about computers and hopefully knows what to do with the mentioned possible causes of the problem.

@manawydan - black outs could be power supply, but I still suspect its the graphics adapter (aka video cards), its the thing that puts the images onto the screen.

I wouldn't be surprised if ZT exercises the graphics in 'unusual ways'. I don't know why it is, but graphics adapters seem to become flaky ie they work most of time but then some weird combination causes them to misbehave.

BR

theducks · 05-12-2014, 10:42 AM

No warning shutdowns can be Power (over/Under rails) related. Bulging caps on the motherboard are common.

While these can be age related, they can also happen on NEW systems (poor quality control by the cap maker. Wrong ESR rating chosen.)

A visual inspection will usually suffice. if the disc end with the X is domed

Severe thermal rise (eg loose heat sink) will also cause a fast power off

manawydan · 05-12-2014, 01:47 PM

There seem to be plenty of possible causes. :/

I have now copied a part of the library to the laptop and used the library maintenance - restore option.

Worked well except this error message:

Spoiler:

Is it just a damaged (?) db entry that causes a problem here or something more severe?

It is Calibre Portable btw.
OS is Win7, 64bit.

P.S. No problems or crashes or anything and Calibre shows up fast when opened.
Will have to see what happens with increasing memory load and runtime but it already is much better than it ever was on the old system.

manawydan · 05-19-2014, 02:08 PM

I am fed up.

It all worked so nicely and then Calibre started to freeze.

Windows 7 64bit, Fujitsu Lifebook laptop, Intel Core i5-3230M, 2,6GHz, 4GB RAM, 750GB HDD, Intel HD.
Calibre portable 1.33.

I copied the old library over but it was not possible to simply rename and use it as Calibre Library or to switch to it.

Well, I started to move folders into the actual library and asked Calibre to restore the db.
That worked.

Repeated it with another bunch and so on.
I removed settings folder first, not sure if this was necessary though.

But then it started to show error messages. Mainly something like "calibre.exe stopped" etc.

I even tried to run it with Zoomtext disabled. No help.
I am really stuck.
The hardware is ok, the OS is new, up until 30-40000 books it went ok.

I just don't get it.

04-20-2014, 06:37 PM	#1
manawydan Connoisseur Posts: 95 Karma: 5854 Join Date: Aug 2011 Device: none	How to best start db from scratch? After my old db (or Calibre, I don't know) acts more and more funny I have decided to start from scratch. On a laptop, Windows 7 OS, wit the newest Calibre version. My question is how to best do it? What would you recommend? If I simply copy the old library folder - at once or step by step - over but not the .db and .json file (these are provided by the fresh installed Calibre) and then point Calibre to the folder with the copied library would this prevent problems that could mess up the database? It is a big library with 100K+ database etnries.

04-25-2014, 07:14 PM	#7
theducks Well trained by Cats Posts: 31,047 Karma: 60358908 Join Date: Aug 2009 Location: The Central Coast of California Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A	+3 You have a basic, @ SYSTEM level failure . Hardware dying, includes poor cooling or power rails failing. You have a Malware infestation A wrong driver was installed during an update.. (Admin level logon required) You should look at all RED items for clues, Review yellow items as some might be a warning of impending Hardware demise (I caught a HD going bad that way, while it was good enough to clone and go ) Look at the Event viewer. All Programs: Administrative Tools: Event viewer (Windows) System Log:

05-12-2014, 01:47 PM	#14
manawydan Connoisseur Posts: 95 Karma: 5854 Join Date: Aug 2011 Device: none	There seem to be plenty of possible causes. :/ I have now copied a part of the library to the laptop and used the library maintenance - restore option. Worked well except this error message: Spoiler: calibre, version 1.33.0 WARNING: Success: Restoring the database succeeded with some warnings. Click "Show details" to see the details. The old database was saved as: <snip> Failed to restore the books in the following folders: C:\Calibre Portable\Calibre Library\Andrew Lane\Young Sherlock Holmes- Death Cloud (19078) with error: Traceback (most recent call last): File "site-packages\calibre\db\restore.py", line 253, in restore_books File "site-packages\calibre\db\cache.py", line 57, in call_func_with_lock File "site-packages\calibre\db\cache.py", line 1787, in restore_book File "site-packages\calibre\db\backend.py", line 1668, in restore_book File "site-packages\calibre\db\backend.py", line 803, in executemany File "c:\cygwin\home\kovid\sw\build\apsw-3.7.17-r1\src\cursor.c", line 231, in resetcursor ConstraintError: ConstraintError: columns book, format are not unique Is it just a damaged (?) db entry that causes a problem here or something more severe? It is Calibre Portable btw. OS is Win7, 64bit. P.S. No problems or crashes or anything and Calibre shows up fast when opened. Will have to see what happens with increasing memory load and runtime but it already is much better than it ever was on the old system.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Calibre wont start (first time start up)	Mike Railton	Devices	15	05-25-2012 02:14 AM
Reflash K3 from scratch?	rapidblue	Kindle Developer's Corner	5	11-01-2010 01:57 PM
PRS-300 What's the easiest way to start from scratch?	Angie	Sony Reader	2	09-16-2010 09:01 PM
Starting from Scratch (again!)	krisk	Lounge	55	06-10-2009 06:25 PM
iLiad How to totally clean out and start from scratch?	haven	iRex Developer's Corner	6	02-05-2009 10:31 PM

04-20-2014, 07:25 PM	#2
eschwartz Ex-Helpdesk Junkie Posts: 19,421 Karma: 85400180 Join Date: Nov 2012 Location: The Beaten Path, USA, Roundworld, This Side of Infinity Device: Kindle Touch fw5.3.7 (Wifi only)	calibre has a built-in utility to "Rebuild Database" from scratch. It is under Library icon ==> Libray Maintenance. It will replace the db with a new one built from the metadata.opf backups, plus the .json with your settings backup.

04-25-2014, 04:12 PM	#5
eschwartz Ex-Helpdesk Junkie Posts: 19,421 Karma: 85400180 Join Date: Nov 2012 Location: The Beaten Path, USA, Roundworld, This Side of Infinity Device: Kindle Touch fw5.3.7 (Wifi only)	Sounds like hardware/system problems, all right. If other programs are crashing too, that is almost a guarantee. The db definitely is not responsible for those kinds of problems. Library Maintenance ==> Library Restore doesn't use the database, AFAIK. That's the whole point of it.

05-02-2014, 07:05 PM	#9
BetterRed null operator (he/him) Posts: 21,721 Karma: 29711016 Join Date: Mar 2012 Location: Sydney Australia Device: none	@manawydan - I wonder of the magnifier puts extra load on the graphics adapter. As you will read in that thread I previously referenced I had some 'weird problems', not with calibre specifically, but generally. The cause was a flaky graphics adapter, but determining that it was the cause was in the end - a hunch. And a couple of friends who have had inexplicable problems solved them by replacing the graphics adapter. BR

05-02-2014, 09:13 PM	#10
Dopedangel Wizard Posts: 1,793 Karma: 30548723 Join Date: Dec 2006 Location: Singapore Device: Boyue	You can copy your library to the new pc. If you are still getting problems use rebuild database on the new PC. I would recommend breaking up your library into smaller libraries generally something not recommended but with such a large library as your you would be better off. BTW if you are using custom columns generated from other columns it will hit your calibre performance quite a bit as you have over 100K entries and they are generated on the fly

05-12-2014, 06:28 AM	#11
manawydan Connoisseur Posts: 95 Karma: 5854 Join Date: Aug 2011 Device: none	Haven't made much progress (no time for dealing with Calibre) but wanted to thank you for the link. The more I read the more I tend to agree that the magnifyer (Zoomtext) + the graphics system are good suspects. I am now really curious how this will work on a Windows 7 64bit system with the newest ZT version. I also noticed that recently there seem to be a black out (as mentioned in the linked posts - no blue scren just a immediate shutdown) when using a backup programme and trying to download a bigger file from a certain website (and only this site). However I managed to do both with ZT switched off and suddenly there were no black outs at all. I am not sure what exactly this means but it's ... interesting. No idea what a graphic adapter actually is or how to change it but I know someone who knows quite a bit about computers and hopefully knows what to do with the mentioned possible causes of the problem.

05-12-2014, 10:42 AM	#13
theducks Well trained by Cats Posts: 31,047 Karma: 60358908 Join Date: Aug 2009 Location: The Central Coast of California Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A	No warning shutdowns can be Power (over/Under rails) related. Bulging caps on the motherboard are common. While these can be age related, they can also happen on NEW systems (poor quality control by the cap maker. Wrong ESR rating chosen.) A visual inspection will usually suffice. if the disc end with the X is domed Severe thermal rise (eg loose heat sink) will also cause a fast power off

05-19-2014, 02:08 PM	#15
manawydan Connoisseur Posts: 95 Karma: 5854 Join Date: Aug 2011 Device: none	I am fed up. It all worked so nicely and then Calibre started to freeze. Windows 7 64bit, Fujitsu Lifebook laptop, Intel Core i5-3230M, 2,6GHz, 4GB RAM, 750GB HDD, Intel HD. Calibre portable 1.33. I copied the old library over but it was not possible to simply rename and use it as Calibre Library or to switch to it. Well, I started to move folders into the actual library and asked Calibre to restore the db. That worked. Repeated it with another bunch and so on. I removed settings folder first, not sure if this was necessary though. But then it started to show error messages. Mainly something like "calibre.exe stopped" etc. I even tried to run it with Zoomtext disabled. No help. I am really stuck. The hardware is ok, the OS is new, up until 30-40000 books it went ok. I just don't get it.

Advert

Advert