Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Library Management

Notices

Reply
 
Thread Tools Search this Thread
Old 08-19-2015, 06:18 AM   #1
BranMakMorn
Enthusiast
BranMakMorn began at the beginning.
 
Posts: 30
Karma: 10
Join Date: Jan 2010
Device: none
Sudden Big Problems with Library

Dear all,

I've used Calibre for years, since 2010 in fact, and I never had a problem with it. I now have a very large library of over 7300 books.

A few days back, all of a sudden, Calibre started giving me a strange error when opening books (double clicking on the title) something like "I/O error".

It eventually told me the metadata database was corrupted and that it needed to re-check everything. So it did, taking a pretty long time.

Now, I got a report with LONG list of "Invalid Titles", "Extra titles", "Extra Authors", "Extra Book Formats" and "Unknown Files in Books" (hundreds of entries).

In practical terms, I have 3 different kind of problems:

1) there are many titles which appear in the database but that tells me it cannot find the file, while in fact the PDF file is where it should be (I click "open path" and there it is) OR there are even TWO copies of the same PDF;

2) there are many titles which appear in the database but that tells me it cannot find the file, and in fact the relevant folder has only the "cover" and the "metadata": where the hell did my books go?? (this is what really worries me).

2) there are many books for which I have the PDF in the correct folder, or again sometimes even two files, or two metadata files, (but sometimes the folder has no number in brackets) which do not appear on the database.

[NB these problems do not match exactly with the categories of error above: if it search for titles in the list of "Extra Titles" sometimes they will not appear in the database but will be in the folder, sometimes it will show in the database and also work fine...]

I don't know what to do. I tried to uninstall and reinstall the lastest version, using the old library, but didn't help. I tried to tell calibre to restore database twice, but I always get error messages. Can anyone hand-hold me through this? I am mostly concerned about the many seemingly missing books. As a LAST resort I could explode the calibre library, delete everything but the PDFs and re-add everything in a new fresh installed program, but that would mean having to re-correct everything for over 7000 books. It took me years to populate and keep this library database tidy.

Help please!

Last edited by BranMakMorn; 08-19-2015 at 06:32 AM.
BranMakMorn is offline   Reply With Quote
Old 08-19-2015, 08:11 AM   #2
itimpi
Wizard
itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.
 
Posts: 4,552
Karma: 950151
Join Date: Nov 2008
Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)
When there is a very large mismatch between what files are in a Calibre library and what Calibre thinks should be there then the easiest way to recover is typically something along the lines of:
  • Use the Calibre facility to move the library to a new location. This will move all the files/folders that Calibre knows about to the new location.
  • Look at what has been left behind with a view to adding back any items that you think should be in your Calibre library
This does not explain why your library seems to have got messed up in the first place. Reasons I have come across are:
  • Storing the Calibre library on Cloud based storage. If you ever run Calibre from two locations at the same time or if the sync process interferes with a running instance of Calibre then library corruption can occur.
  • Storing the Calibre library on a network drive. This works a lot of the time but it is not officially supported as things occasionally go wrong.
  • Storing the Calibre library in Google Drive. This can mess up a Calibre library by silently renaming folders with numbers in brackets (as used by Calibre).
  • Any manual manipulation of the files stored in the Calibre library bypassing the Calibre GUI.
itimpi is offline   Reply With Quote
Advert
Old 08-19-2015, 08:18 AM   #3
BranMakMorn
Enthusiast
BranMakMorn began at the beginning.
 
Posts: 30
Karma: 10
Join Date: Jan 2010
Device: none
Quote:
Originally Posted by itimpi View Post
When there is a very large mismatch between what files are in a Calibre library and what Calibre thinks should be there then the easiest way to recover is typically something along the lines of:
  • Use the Calibre facility to move the library to a new location. This will move all the files/folders that Calibre knows about to the new location.
  • Look at what has been left behind with a view to adding back any items that you think should be in your Calibre library
This does not explain why your library seems to have got messed up in the first place. Reasons I have come across are:
  • Storing the Calibre library on Cloud based storage. If you ever run Calibre from two locations at the same time or if the sync process interferes with a running instance of Calibre then library corruption can occur.
  • Storing the Calibre library on a network drive. This works a lot of the time but it is not officially supported as things occasionally go wrong.
  • Storing the Calibre library in Google Drive. This can mess up a Calibre library by silently renaming folders with numbers in brackets (as used by Calibre).
  • Any manual manipulation of the files stored in the Calibre library bypassing the Calibre GUI.

Damn, point n3 applies. I keep my Calibre Library inside the Google Drive folder, because 1) I want a constant backup and 2) I read my PDFs on an iPad, and I sync the book I need from Google Drive, do my annotations so that they then are automatically added to the file on my computer. It's strange tho, because I've used this system for years (for about 3 years I was on Sugarsync, then they decided to charge, so I moved to Google Drive about a year ago). Could it be just that?

But again, what mostly concerns me is: were are the 200+ books which now are just missing? if I order my Calibre library for file size I can easily see that I've got over 200 entries without a size. If i click view path some of them will be there (so why doesn't it see them?), but some are missing...
BranMakMorn is offline   Reply With Quote
Old 08-19-2015, 08:32 AM   #4
itimpi
Wizard
itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.
 
Posts: 4,552
Karma: 950151
Join Date: Nov 2008
Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)
Quote:
Originally Posted by BranMakMorn View Post
Damn, point n3 applies. I keep my Calibre Library inside the Google Drive folder, because 1) I want a constant backup and 2) I read my PDFs on an iPad, and I sync the book I need from Google Drive, do my annotations so that they then are automatically added to the file on my computer. It's strange tho, because I've used this system for years (for about 3 years I was on Sugarsync, then they decided to charge, so I moved to Google Drive about a year ago). Could it be just that?

But again, what mostly concerns me is: were are the 200+ books which now are just missing? if I order my Calibre library for file size I can easily see that I've got over 200 entries without a size. If i click view path some of them will be there (so why doesn't it see them?), but some are missing...
The problem is that under certain circumstance Google Drive will remove the number in brackets in the folders in the Calibre library. I think if you only ever sync one way things tend to be OK, but as soon as you start two-way sync then problems start to occur. I believe Google Drive expects numbers in brackets to be used for its own versioning system. In such circumstances when Google Drive removes or changes the number in brackets, Calibre loses track of the files that were in that folder. This may be what you are seeing in that the books are still on the disk but not in the folder where Calibre thinks they should be stored.

The process I mentioned should allow you to find and rectify such problems. I think someone also put together a program which could scan a Calibre library stored on Google Drive and correct cases where the number in brackets had been removed, but I do not have any link to it.

If you want to use a Cloud service for your Calibre library then something like Copy.com or OneDrive might be better choices. Note, however, that when using Cloud Services you must always make certain that you never have two instances of Calibre at different locations running against the same Calibre library or corruption will occur as Calibre is not multi-user aware in its current incarnation).
itimpi is offline   Reply With Quote
Old 08-19-2015, 08:45 AM   #5
BranMakMorn
Enthusiast
BranMakMorn began at the beginning.
 
Posts: 30
Karma: 10
Join Date: Jan 2010
Device: none
Quote:
Originally Posted by itimpi View Post
The problem is that under certain circumstance Google Drive will remove the number in brackets in the folders in the Calibre library. I think if you only ever sync one way things tend to be OK, but as soon as you start two-way sync then problems start to occur. I believe Google Drive expects numbers in brackets to be used for its own versioning system. In such circumstances when Google Drive removes or changes the number in brackets, Calibre loses track of the files that were in that folder. This may be what you are seeing in that the books are still on the disk but not in the folder where Calibre thinks they should be stored.

The process I mentioned should allow you to find and rectify such problems. I think someone also put together a program which could scan a Calibre library stored on Google Drive and correct cases where the number in brackets had been removed, but I do not have any link to it.

If you want to use a Cloud service for your Calibre library then something like Copy.com or OneDrive might be better choices. Note, however, that when using Cloud Services you must always make certain that you never have two instances of Calibre at different locations running against the same Calibre library or corruption will occur as Calibre is not multi-user aware in its current incarnation).

Thanks, I'll try to follow the process you described. And no, I never run Calibre on any other computer but mine. One last question though: why do I find in so many cases that a given book has been duplicated in the folder, so that I see, for ex, "How to Fix your Calibre - John Doe" AND "How to Fix your Calibre - John Doe (1)" ?
BranMakMorn is offline   Reply With Quote
Advert
Old 08-19-2015, 09:07 AM   #6
itimpi
Wizard
itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.
 
Posts: 4,552
Karma: 950151
Join Date: Nov 2008
Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)
Quote:
Originally Posted by BranMakMorn View Post
Thanks, I'll try to follow the process you described. And no, I never run Calibre on any other computer but mine. One last question though: why do I find in so many cases that a given book has been duplicated in the folder, so that I see, for ex, "How to Fix your Calibre - John Doe" AND "How to Fix your Calibre - John Doe (1)" ?
Not sure of this? Are you talking about the folder holding the book or the actual ebook files? I suspect it is something that Google drive is doing.

Points to note
  • Calibre book folders will always have a number in brackets. The number is normally large as it is actually a row id from the books table in the Calibre metadata.db database. It should also be different for every ebook folder in the library.
  • Calibre ebook files will never have a number in brackets at the end. If you are seeing such files then I expect they are being created by Google Drive for some reason.
itimpi is offline   Reply With Quote
Old 08-19-2015, 09:58 AM   #7
theducks
Well trained by Cats
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 29,659
Karma: 54369090
Join Date: Aug 2009
Location: The Central Coast of California
Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A
Quote:
Originally Posted by BranMakMorn View Post
Dear all,

I've used Calibre for years, since 2010 in fact, and I never had a problem with it. I now have a very large library of over 7300 books.

A few days back, all of a sudden, Calibre started giving me a strange error when opening books (double clicking on the title) something like "I/O error".

It eventually told me the metadata database was corrupted and that it needed to re-check everything. So it did, taking a pretty long time.
Help please!
STOP do NOT attempt to repair at this time.
Backup your Library (and settings)

IO error on the main library is almost always a drive (or hardware) failure

any attempt to repair the Drives file system is like putting a person having a Heart Attach, on a treadmill.

Find and fix the IO problem: (2010 hardware is Old age hardware. Consumer grade does not always last as long as business grade)

If your library is on a Flash drive, replace the drive. They have finite cycles.
If your drive is a external USB HD:
1) make sure the power supply is firmly connected (IMHO self (USB) powered drives are marginal designs).
2) Replace the USB cable.
3) Verify the port connector is still firm. Better yet. change ports
theducks is offline   Reply With Quote
Old 08-19-2015, 10:03 AM   #8
eschwartz
Ex-Helpdesk Junkie
eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.
 
eschwartz's Avatar
 
Posts: 19,422
Karma: 85397180
Join Date: Nov 2012
Location: The Beaten Path, USA, Roundworld, This Side of Infinity
Device: Kindle Touch fw5.3.7 (Wifi only)
See this sticky in the Related Tools subforum: WARNING - Google Drive Incompatible with Calibre!

It has links to the two tools developed at various times by MR members to fix such a library.
eschwartz is offline   Reply With Quote
Old 08-19-2015, 10:28 AM   #9
BranMakMorn
Enthusiast
BranMakMorn began at the beginning.
 
Posts: 30
Karma: 10
Join Date: Jan 2010
Device: none
Umm, ok so just to be clear: is it Google Drive or is my HDD failing? It's on no external of flash drive, it's my laptop HDD. I also always backup the whole thing on an external one, like every 2 months, and I've done it yesterday.

EDIT: and my laptop is not even 2 years old.

Last edited by BranMakMorn; 08-19-2015 at 10:31 AM.
BranMakMorn is offline   Reply With Quote
Old 08-19-2015, 10:55 AM   #10
PeterT
Grand Sorcerer
PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.
 
PeterT's Avatar
 
Posts: 12,103
Karma: 73447988
Join Date: Nov 2007
Location: Toronto
Device: Nexus 7, Clara, Touch, Tolino EPOS
Take a look in the Windows Event Viewer and see if there are any events related to disk failure.
PeterT is offline   Reply With Quote
Old 08-19-2015, 11:12 AM   #11
eschwartz
Ex-Helpdesk Junkie
eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.
 
eschwartz's Avatar
 
Posts: 19,422
Karma: 85397180
Join Date: Nov 2012
Location: The Beaten Path, USA, Roundworld, This Side of Infinity
Device: Kindle Touch fw5.3.7 (Wifi only)
Quote:
Originally Posted by BranMakMorn View Post
Umm, ok so just to be clear: is it Google Drive or is my HDD failing? It's on no external of flash drive, it's my laptop HDD. I also always backup the whole thing on an external one, like every 2 months, and I've done it yesterday.

EDIT: and my laptop is not even 2 years old.
I believe theducks is being overcautious because of the IOError. But an IOError could simply mean the file isn't on disk...

All your signs point to a Google Drive attack, I personally see no reason to suspect HDD failure.
(But it never hurts to check a drive for hardware failures.)
eschwartz is offline   Reply With Quote
Old 08-19-2015, 12:56 PM   #12
theducks
Well trained by Cats
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 29,659
Karma: 54369090
Join Date: Aug 2009
Location: The Central Coast of California
Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A
All of a sudden, 'Google Drive' entered this discussion.

To be clear: Do you use Google drive for ANYTHING?
If the answer is NO, then the 'Google drive' is a red herring (In you case)
GD is a known Library issue

The event viewer (system section) is a good tip (look at red items)
theducks is offline   Reply With Quote
Old 08-19-2015, 01:09 PM   #13
BranMakMorn
Enthusiast
BranMakMorn began at the beginning.
 
Posts: 30
Karma: 10
Join Date: Jan 2010
Device: none
So to be clear:

YES, I keep my entire Calibre Library folder inside my Google Drive folder.
NO, to the best of my understanding, Event Viewer has no Disk-related errors.

So, I suppose it's just Google Drive's fault. Which sucks, but I guess I can look into OneDrive.

Tomorrow morning I'll try to apply the fix offered above.
BranMakMorn is offline   Reply With Quote
Old 08-19-2015, 01:25 PM   #14
PeterT
Grand Sorcerer
PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.
 
PeterT's Avatar
 
Posts: 12,103
Karma: 73447988
Join Date: Nov 2007
Location: Toronto
Device: Nexus 7, Clara, Touch, Tolino EPOS
What many people (ok; some people (ok; I know I do)) is keep their library OUTSIDE of any of the Google Drive / Dropbox / ... folders on their PC, but then use a tool to do a ONE WAY sync from the location on their PC to the Google Drive / Dropbox / ... folder on their PC.

In my case I use Allway Sync http://allwaysync.com/
PeterT is offline   Reply With Quote
Old 08-19-2015, 01:33 PM   #15
theducks
Well trained by Cats
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 29,659
Karma: 54369090
Join Date: Aug 2009
Location: The Central Coast of California
Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A
Quote:
Originally Posted by BranMakMorn View Post
So to be clear:

YES, I keep my entire Calibre Library folder inside my Google Drive folder.
NO, to the best of my understanding, Event Viewer has no Disk-related errors.

So, I suppose it's just Google Drive's fault. Which sucks, but I guess I can look into OneDrive.

Tomorrow morning I'll try to apply the fix offered above.
Thanks for clearing that up.

GD is just not a good fit with the Calibre Library structure
Whatever you use. Pause synch when using Calibre. Things happen in rapid sequences. 1 way (as a backup only) is better for the same reason
theducks is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Big Library Read Fbone General Discussions 7 06-09-2015 06:02 PM
Big Library (accelerate starting?) epubfan Library Management 1 12-28-2013 02:58 PM
Sudden increase in problems Darnielle Kobo Tablets 29 12-10-2011 08:02 AM
Sudden big slowdown in page turn time KlondikeGeoff Sony Reader 16 06-25-2011 10:15 PM
New Owner - Big Problems Crumbz iRex 9 11-04-2008 12:40 AM


All times are GMT -4. The time now is 11:51 PM.


MobileRead.com is a privately owned, operated and funded community.