Register Guidelines E-Books Today's Posts Search

Go Back   MobileRead Forums > E-Book Software > Calibre > Library Management

Notices

Reply
 
Thread Tools Search this Thread
Old 04-20-2014, 06:37 PM   #1
manawydan
Connoisseur
manawydan shares his or her toysmanawydan shares his or her toysmanawydan shares his or her toysmanawydan shares his or her toysmanawydan shares his or her toysmanawydan shares his or her toysmanawydan shares his or her toysmanawydan shares his or her toysmanawydan shares his or her toysmanawydan shares his or her toysmanawydan shares his or her toys
 
Posts: 95
Karma: 5854
Join Date: Aug 2011
Device: none
How to best start db from scratch?

After my old db (or Calibre, I don't know) acts more and more funny I have decided to start from scratch.
On a laptop, Windows 7 OS, wit the newest Calibre version.

My question is how to best do it?
What would you recommend?

If I simply copy the old library folder - at once or step by step - over but not the .db and .json file (these are provided by the fresh installed Calibre) and then point Calibre to the folder with the copied library would this prevent problems that could mess up the database?

It is a big library with 100K+ database etnries.
manawydan is offline   Reply With Quote
Old 04-20-2014, 07:25 PM   #2
eschwartz
Ex-Helpdesk Junkie
eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.
 
eschwartz's Avatar
 
Posts: 19,421
Karma: 85400180
Join Date: Nov 2012
Location: The Beaten Path, USA, Roundworld, This Side of Infinity
Device: Kindle Touch fw5.3.7 (Wifi only)
calibre has a built-in utility to "Rebuild Database" from scratch. It is under Library icon ==> Libray Maintenance.

It will replace the db with a new one built from the metadata.opf backups, plus the .json with your settings backup.
eschwartz is offline   Reply With Quote
Advert
Old 04-20-2014, 07:37 PM   #3
BetterRed
null operator (he/him)
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 21,699
Karma: 29711016
Join Date: Mar 2012
Location: Sydney Australia
Device: none
Quote:
Originally Posted by manawydan View Post
After my old db (or Calibre, I don't know) acts more and more funny I have decided to start from scratch.
On a laptop, Windows 7 OS, wit the newest Calibre version.

My question is how to best do it?
What would you recommend?

If I simply copy the old library folder - at once or step by step - over but not the .db and .json file (these are provided by the fresh installed Calibre) and then point Calibre to the folder with the copied library would this prevent problems that could mess up the database?

It is a big library with 100K+ database etnries.
@manawydan - I think you would need to rebuild the database via Library Maintenance->Library restore - that could take a quite a long time (overnight) if "100K+ database entries" means 100,000 'books'. The database will be rebuilt from the .opf files in the book files. If the database is the source of your problems, then rebuilding it from the .opf could replicate the problems.

Have you tried rebuilding the database on the current library, if you have and it didn't fix the problems, then I doubt that copying the author and book folders elsewhere and doing the same thing will resolve your issues - unless you have hardware problems.

It might be better to say what you mean by "my old db... acts more and more funny" - it could be the configuration data that's gone skew-whiff, rebuilding the database wont fix those problems.

Try renaming the config folder (with calibre not running) and then start calibre (it will build a fresh config folder) and point it to the existing library. You'll not have most customisations but it should be 'sufficient' to give you an indication of whether it is the library database that's causing your problems.

You can restore the old config by deleting the 'fresh' one and re-renaming the old one.

BR

Last edited by BetterRed; 04-20-2014 at 07:41 PM.
BetterRed is offline   Reply With Quote
Old 04-25-2014, 04:01 PM   #4
manawydan
Connoisseur
manawydan shares his or her toysmanawydan shares his or her toysmanawydan shares his or her toysmanawydan shares his or her toysmanawydan shares his or her toysmanawydan shares his or her toysmanawydan shares his or her toysmanawydan shares his or her toysmanawydan shares his or her toysmanawydan shares his or her toysmanawydan shares his or her toys
 
Posts: 95
Karma: 5854
Join Date: Aug 2011
Device: none
Quote:
Originally Posted by BetterRed View Post
Have you tried rebuilding the database on the current library, if you have and it didn't fix the problems,
That is not possible.
Calibre can not handle any form of maintenance. It freezes and/or crashes.

To make a long story short, it is possible that Calibre is fine and it is a hardware problem or a hardware plus OS problem or a problem with a screen magnifyer or all together or nothing ..
I can not test it without the magnifyer anyway.

On the laptop there is Windows 7 and a 64bit newest version of this software.

Calibre is not the only software that has problems but with Calibre the problems are the obvious.

It does not have to be the .db - the newest trick are memory errors (though the only changes from when there were none to now is the amount of entries, sometimes the requested operation (e.g. sort, search etc.) will be executed anyway and sometimes I have to completely close the programme and start again.

However, this is a long and complicated story.

My hope now is that if I start from scratch on a new system I can avoid problems or at least see when they start to show up.

Quote:
Try renaming the config folder (with calibre not running) and then start calibre (it will build a fresh config folder)
You mean the setting folder (running Calibre Portable)?
And I leave the library as it is?
I can try but I am quite sure it will not work and Calibre will crash again.

But I'll give it a try, thanks.

Last edited by manawydan; 04-25-2014 at 04:04 PM.
manawydan is offline   Reply With Quote
Old 04-25-2014, 04:12 PM   #5
eschwartz
Ex-Helpdesk Junkie
eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.
 
eschwartz's Avatar
 
Posts: 19,421
Karma: 85400180
Join Date: Nov 2012
Location: The Beaten Path, USA, Roundworld, This Side of Infinity
Device: Kindle Touch fw5.3.7 (Wifi only)
Sounds like hardware/system problems, all right. If other programs are crashing too, that is almost a guarantee.
The db definitely is not responsible for those kinds of problems. Library Maintenance ==> Library Restore doesn't use the database, AFAIK. That's the whole point of it.
eschwartz is offline   Reply With Quote
Advert
Old 04-25-2014, 06:30 PM   #6
BetterRed
null operator (he/him)
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 21,699
Karma: 29711016
Join Date: Mar 2012
Location: Sydney Australia
Device: none
Quote:
Originally Posted by manawydan View Post
You mean the setting folder (running Calibre Portable)?
And I leave the library as it is?
I can try but I am quite sure it will not work and Calibre will crash again.
Yes that's what a I meant - but I tend to agree with eschwartz, if other programs are misbehaving and you're getting memory faults then you probably have a hardware problem.

There's some discussion in this thread that is tangentially germane https://www.mobileread.com/forums/sho...light=graphics - in particular see chaley's comments towards the end of the thread where he discusses why different software may 'exercise' computer components in different ways etc.

I would run some memory tests. Also on laptops there is the problem of heat, the airflow ducts can become clogged quite easily, memory chips (main and video) are very susceptible to heat damage.

BR
BetterRed is offline   Reply With Quote
Old 04-25-2014, 07:14 PM   #7
theducks
Well trained by Cats
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 31,008
Karma: 60358908
Join Date: Aug 2009
Location: The Central Coast of California
Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A
+3

You have a basic, @ SYSTEM level failure .
  • Hardware dying, includes poor cooling or power rails failing.
  • You have a Malware infestation
  • A wrong driver was installed during an update..

(Admin level logon required) You should look at all RED items for clues, Review yellow items as some might be a warning of impending Hardware demise (I caught a HD going bad that way, while it was good enough to clone and go )

Look at the Event viewer. All Programs: Administrative Tools: Event viewer

(Windows) System Log:
theducks is offline   Reply With Quote
Old 05-02-2014, 10:38 AM   #8
manawydan
Connoisseur
manawydan shares his or her toysmanawydan shares his or her toysmanawydan shares his or her toysmanawydan shares his or her toysmanawydan shares his or her toysmanawydan shares his or her toysmanawydan shares his or her toysmanawydan shares his or her toysmanawydan shares his or her toysmanawydan shares his or her toysmanawydan shares his or her toys
 
Posts: 95
Karma: 5854
Join Date: Aug 2011
Device: none
Quote:
Originally Posted by BetterRed View Post
Yes that's what a I meant - but I tend to agree with eschwartz, if other programs are misbehaving and you're getting memory faults then you probably have a hardware problem.
Yes unfortunately.

The PC is not that old though.
It is just not easy right now to find out thre reason.

Quote:
Also on laptops there is the problem of heat, the airflow ducts can become clogged quite easily, memory chips (main and video) are very susceptible to heat damage.
Yes I am aware of the heat problem.
I want to put it on the laptop as alternative until the PC problem is solved (if ever).
I can still use the PC-located Calibre but it is a pain in the ****.

The problems I now have are not on the laptop but the PC.

Cooling can't be a reason because I took good care to not only use better vents than came with the original frame but also get enough built in. Not that I know much about this but I got some advice from people who know about this.

Well of course there can always something go wrong.
And there can be plenty of other possible reasons.

Anyway, I will check out the link and thanks for your help.
manawydan is offline   Reply With Quote
Old 05-02-2014, 07:05 PM   #9
BetterRed
null operator (he/him)
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 21,699
Karma: 29711016
Join Date: Mar 2012
Location: Sydney Australia
Device: none
@manawydan - I wonder of the magnifier puts extra load on the graphics adapter. As you will read in that thread I previously referenced I had some 'weird problems', not with calibre specifically, but generally. The cause was a flaky graphics adapter, but determining that it was the cause was in the end - a hunch. And a couple of friends who have had inexplicable problems solved them by replacing the graphics adapter.

BR
BetterRed is offline   Reply With Quote
Old 05-02-2014, 09:13 PM   #10
Dopedangel
Wizard
Dopedangel ought to be getting tired of karma fortunes by now.Dopedangel ought to be getting tired of karma fortunes by now.Dopedangel ought to be getting tired of karma fortunes by now.Dopedangel ought to be getting tired of karma fortunes by now.Dopedangel ought to be getting tired of karma fortunes by now.Dopedangel ought to be getting tired of karma fortunes by now.Dopedangel ought to be getting tired of karma fortunes by now.Dopedangel ought to be getting tired of karma fortunes by now.Dopedangel ought to be getting tired of karma fortunes by now.Dopedangel ought to be getting tired of karma fortunes by now.Dopedangel ought to be getting tired of karma fortunes by now.
 
Dopedangel's Avatar
 
Posts: 1,792
Karma: 30548723
Join Date: Dec 2006
Location: Singapore
Device: Boyue
You can copy your library to the new pc. If you are still getting problems use rebuild database on the new PC. I would recommend breaking up your library into smaller libraries generally something not recommended but with such a large library as your you would be better off.

BTW if you are using custom columns generated from other columns it will hit your calibre performance quite a bit as you have over 100K entries and they are generated on the fly
Dopedangel is offline   Reply With Quote
Old 05-12-2014, 06:28 AM   #11
manawydan
Connoisseur
manawydan shares his or her toysmanawydan shares his or her toysmanawydan shares his or her toysmanawydan shares his or her toysmanawydan shares his or her toysmanawydan shares his or her toysmanawydan shares his or her toysmanawydan shares his or her toysmanawydan shares his or her toysmanawydan shares his or her toysmanawydan shares his or her toys
 
Posts: 95
Karma: 5854
Join Date: Aug 2011
Device: none
Haven't made much progress (no time for dealing with Calibre) but wanted to thank you for the link.

The more I read the more I tend to agree that the magnifyer (Zoomtext) + the graphics system are good suspects.

I am now really curious how this will work on a Windows 7 64bit system with the newest ZT version.

I also noticed that recently there seem to be a black out (as mentioned in the linked posts - no blue scren just a immediate shutdown) when using a backup programme and trying to download a bigger file from a certain website (and only this site). However I managed to do both with ZT switched off and suddenly there were no black outs at all.

I am not sure what exactly this means but it's ... interesting.

No idea what a graphic adapter actually is or how to change it but I know someone who knows quite a bit about computers and hopefully knows what to do with the mentioned possible causes of the problem.
manawydan is offline   Reply With Quote
Old 05-12-2014, 07:55 AM   #12
BetterRed
null operator (he/him)
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 21,699
Karma: 29711016
Join Date: Mar 2012
Location: Sydney Australia
Device: none
Quote:
Originally Posted by manawydan View Post
Haven't made much progress (no time for dealing with Calibre) but wanted to thank you for the link.

The more I read the more I tend to agree that the magnifyer (Zoomtext) + the graphics system are good suspects.

I am now really curious how this will work on a Windows 7 64bit system with the newest ZT version.

I also noticed that recently there seem to be a black out (as mentioned in the linked posts - no blue scren just a immediate shutdown) when using a backup programme and trying to download a bigger file from a certain website (and only this site). However I managed to do both with ZT switched off and suddenly there were no black outs at all.

I am not sure what exactly this means but it's ... interesting.

No idea what a graphic adapter actually is or how to change it but I know someone who knows quite a bit about computers and hopefully knows what to do with the mentioned possible causes of the problem.
@manawydan - black outs could be power supply, but I still suspect its the graphics adapter (aka video cards), its the thing that puts the images onto the screen.

I wouldn't be surprised if ZT exercises the graphics in 'unusual ways'. I don't know why it is, but graphics adapters seem to become flaky ie they work most of time but then some weird combination causes them to misbehave.

BR
BetterRed is offline   Reply With Quote
Old 05-12-2014, 10:42 AM   #13
theducks
Well trained by Cats
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 31,008
Karma: 60358908
Join Date: Aug 2009
Location: The Central Coast of California
Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A
No warning shutdowns can be Power (over/Under rails) related. Bulging caps on the motherboard are common.

While these can be age related, they can also happen on NEW systems (poor quality control by the cap maker. Wrong ESR rating chosen.)

A visual inspection will usually suffice. if the disc end with the X is domed

Severe thermal rise (eg loose heat sink) will also cause a fast power off
theducks is offline   Reply With Quote
Old 05-12-2014, 01:47 PM   #14
manawydan
Connoisseur
manawydan shares his or her toysmanawydan shares his or her toysmanawydan shares his or her toysmanawydan shares his or her toysmanawydan shares his or her toysmanawydan shares his or her toysmanawydan shares his or her toysmanawydan shares his or her toysmanawydan shares his or her toysmanawydan shares his or her toysmanawydan shares his or her toys
 
Posts: 95
Karma: 5854
Join Date: Aug 2011
Device: none
There seem to be plenty of possible causes. :/

I have now copied a part of the library to the laptop and used the library maintenance - restore option.

Worked well except this error message:

Spoiler:

calibre, version 1.33.0
WARNING: Success: Restoring the database succeeded with some warnings. Click "Show details" to see the details. The old database was saved as: <snip>


Failed to restore the books in the following folders:
C:\Calibre Portable\Calibre Library\Andrew Lane\Young Sherlock Holmes- Death Cloud (19078) with error:
Traceback (most recent call last):
File "site-packages\calibre\db\restore.py", line 253, in restore_books
File "site-packages\calibre\db\cache.py", line 57, in call_func_with_lock
File "site-packages\calibre\db\cache.py", line 1787, in restore_book
File "site-packages\calibre\db\backend.py", line 1668, in restore_book
File "site-packages\calibre\db\backend.py", line 803, in executemany
File "c:\cygwin\home\kovid\sw\build\apsw-3.7.17-r1\src\cursor.c", line 231, in resetcursor
ConstraintError: ConstraintError: columns book, format are not unique

Is it just a damaged (?) db entry that causes a problem here or something more severe?

It is Calibre Portable btw.
OS is Win7, 64bit.

P.S. No problems or crashes or anything and Calibre shows up fast when opened.
Will have to see what happens with increasing memory load and runtime but it already is much better than it ever was on the old system.
manawydan is offline   Reply With Quote
Old 05-19-2014, 02:08 PM   #15
manawydan
Connoisseur
manawydan shares his or her toysmanawydan shares his or her toysmanawydan shares his or her toysmanawydan shares his or her toysmanawydan shares his or her toysmanawydan shares his or her toysmanawydan shares his or her toysmanawydan shares his or her toysmanawydan shares his or her toysmanawydan shares his or her toysmanawydan shares his or her toys
 
Posts: 95
Karma: 5854
Join Date: Aug 2011
Device: none
I am fed up.

It all worked so nicely and then Calibre started to freeze.

Windows 7 64bit, Fujitsu Lifebook laptop, Intel Core i5-3230M, 2,6GHz, 4GB RAM, 750GB HDD, Intel HD.
Calibre portable 1.33.

I copied the old library over but it was not possible to simply rename and use it as Calibre Library or to switch to it.

Well, I started to move folders into the actual library and asked Calibre to restore the db.
That worked.

Repeated it with another bunch and so on.
I removed settings folder first, not sure if this was necessary though.

But then it started to show error messages. Mainly something like "calibre.exe stopped" etc.

I even tried to run it with Zoomtext disabled. No help.
I am really stuck.
The hardware is ok, the OS is new, up until 30-40000 books it went ok.

I just don't get it.
manawydan is offline   Reply With Quote
Reply


Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Calibre wont start (first time start up) Mike Railton Devices 15 05-25-2012 02:14 AM
Reflash K3 from scratch? rapidblue Kindle Developer's Corner 5 11-01-2010 01:57 PM
PRS-300 What's the easiest way to start from scratch? *Angie* Sony Reader 2 09-16-2010 09:01 PM
Starting from Scratch (again!) krisk Lounge 55 06-10-2009 06:25 PM
iLiad How to totally clean out and start from scratch? haven iRex Developer's Corner 6 02-05-2009 10:31 PM


All times are GMT -4. The time now is 12:14 AM.


MobileRead.com is a privately owned, operated and funded community.