bug report and associated questions

[paulrw]

Is this the correct place for this?
Yesterday I set up a new library and imported 60ish books to it, edited metadata, covers etc. converted to ePub, and checked all was well. Restarted my PC for updates and the library had disappeared completely - can't find it on disc (I didn't actually check that Calibre had put it where I thought I had told it to when I was working on it), and now when I click the Library icon I get

ERROR: ERROR: Unhandled exception: <b>ImportError</b>:Bad magic number in C:\Program Files\Calibre2\Lib\site-packages\calibre\gui2\dialogs\choose_library_ui.py o

Traceback (most recent call last):
File "site-packages\calibre\gui2\layout.py", line 420, in choose_library
File "site-packages\calibre\gui2\dialogs\choose_library.py", line 12, in <module>
ImportError: Bad magic number in C:\Program Files\Calibre2\Lib\site-packages\calibre\gui2\dialogs\choose_library_ui.py o

Following my previous experience I have checked the wastebasket in case Calibre had arbitrarily decided to to delete the folder, but it is not there.
I Love Calibre's functionality, but am slightly frightened by how easily it seems to lose books. I also had an experience yesterday where my library (before creation of second library) was showing 534 books, I added another 20 and it showed 515! I haven't yet tracked down whether that is genuine or not - one of the main reasons for Calibre is that I have ebooks in a lot of different places so it is not easy to check the Calibre library against the original sources.
One thing I have noticed is that there are quite a lot of duplicate folders for authors, one of which is always empty. If I could get the library function working again I was thinking of creating a new library and importing from the folder structure of the old one to clean it up - would this be sensible?

[DoctorOhh]

Originally Posted by paulrw

Is this the correct place for this?

Yes

Originally Posted by paulrw

I Love Calibre's functionality, but am slightly frightened by how easily it seems to lose books.

I feel your pain, but it is remarkably difficult to tumble upon the keystrokes needed for calibre to lose books. Consider that a million plus folks currently use calibre... Now consider how many folks report losing books...

Originally Posted by paulrw

I also had an experience yesterday where my library (before creation of second library) was showing 534 books, I added another 20 and it showed 515!

Hopefully someone here will help you figure out what is happening before you proceed further.

Originally Posted by paulrw

One thing I have noticed is that there are quite a lot of duplicate folders for authors, one of which is always empty.

That is not the norm for this software. When initially pointing to a library for calibre the folder should be empty. If you pointed to a folder structure that already had books it can get confusing.

Originally Posted by paulrw

If I could get the library function working again I was thinking of creating a new library and importing from the folder structure of the old one to clean it up - would this be sensible?

You can import from the folder structure but most of your changes to metadata would have been in the metadata.db file and not in the books themselves.

What version of calibre are you using?

If you just upgraded, from which version did you upgrade?

Do you backup your library and metadata.db file on a regular schedule?

Some backup strategies are discussed here.

Good Luck

[Worldwalker]

I've been using calibre since 6.41 and I've never had it mislay even a single ebook out of the 2500 or so I have in my library, so I can't agree with you that it "easily" loses books.

If you're mucking around in calibre's private library files, this may have something to do with your vanishing books issue. Don't try to "clean it up"; just let it alone (read my .sig).

As for the proper place for bug reports, that's kind of complicated. If it's something that may be a user issue -- that is, if you've broken something -- you can usually get help/answers here. However, if it's a genuine calibre bug, you need to open a ticket with the bug tracker so it can get fixed.

[paulrw]

I've been using calibre since 6.41 and I've never had it mislay even a single ebook out of the 2500 or so I have in my library, so I can't agree with you that it "easily" loses books.

I have been using it for 2 weeks and this is the third occasion I appear to have lost books!

If you're mucking around in calibre's private library files, this may have something to do with your vanishing books issue. Don't try to "clean it up"; just let it alone (read my .sig).

I have only been into the folders to try to find books after they were lost, not before.

As for the proper place for bug reports, that's kind of complicated.

So is my error message something that needs a ticket?

[paulrw]

What version of calibre are you using?

Latest version 0.7.10, started on 0.7.09. On reflection, the calibre update could have been in the middle of all this.

Do you backup your library and metadata.db file on a regular schedule?

Not at the moment, I am still (trying) to build my library - I have something over 2000 books to import so regular backups while I am doing it are a bit difficult. I was thinking of picking up a Python manual and knocking together a backup facility with the 7-Zip libraries.

[DoctorOhh]

Originally Posted by paulrw

Latest version 0.7.10, started on 0.7.09. On reflection, the calibre update could have been in the middle of all this.

Not at the moment, I am still (trying) to build my library - I have something over 2000 books to import so regular backups while I am doing it are a bit difficult. I was thinking of picking up a Python manual and knocking together a backup facility with the 7-Zip libraries.

Hopefully Kovid will comment later today. If he doesn't then you may have to open a ticket to get his assistance. What OS are you running? You might have to describe your setup in some detail because the devil is in the details.

You stated previously,

Following my previous experience I have checked the wastebasket in case Calibre had arbitrarily decided to to delete the folder, but it is not there.

The only way any books get deleted and sent to the wastebasket is when you use the remove books feature. Restoring them from the wastebasket will not add them back to your library. Although folders come and go all of the time as the metadata changes the folders/file names change too.

[paulrw]

The only way any books get deleted and sent to the wastebasket is when you use the remove books feature. Restoring them from the wastebasket will not add them back to your library.

This was a previous problem. I had added 3 books, and Calibre told me it had finished, but when I looked the entries were there with no formats against them. I checked the folders and they were empty, so I thought I would delete the entries and re-import them. I selected the 3 entries and deleted them, and found that another 115 books had gone as well - no way I could have selected them by accident.

Although folders come and go all of the time as the metadata changes the folders/file names change too.

In my case folders seem to come more than they go The opposite is true of books

[DoctorOhh]

Originally Posted by paulrw

This was a previous problem. I had added 3 books, and Calibre told me it had finished, but when I looked the entries were there with no formats against them. I checked the folders and they were empty, so I thought I would delete the entries and re-import them. I selected the 3 entries and deleted them, and found that another 115 books had gone as well - no way I could have selected them by accident.

I'm a little confused. Why are you checking folders? How did you determine that the entries had "no formats against them"? You selected 3 entries and deleted them? Were you using a file management tool like explorer in Windows or did you choose remove books via the calibre interface?

[Adoby]

Calibre keeps track of the books and folders using a database. If you add, delete or restore books manually in the Calibre Library folders, and don't update the database, you are bound for trouble. Don't do it!

Treat the whole Calibre Library folder as a singe black-box object.

Use only Add/Remove/Convert from within the program to make sure the database is in sync with the Calibre Library. Othervise you WILL MOST LIKELY loose data and corrupt the Calibre Library, as you have noticed.

This is not a bug.

[paulrw]

Using Calibre, clicked on the entry. in the right hand pane formats: was empty. Using Windows I looked in the directory shown under "Path:" in the right hand pane. The directory was empty. I did this to make sure that Calibre wasn't simply misreporting the fomat. Since there was nothing there, I used the Calibre recycle icon which would I thought remove the entry (I think I did try the Preferences/Advanced/Check Database Integrity first).
I can't see any way that I could have selected the other books, they were not contiguous in the view I had, and whilst I admit my memory is not what it was, I think I would have remembered holding the CTRL key down for 115 clicks.
To clarify things more, I am using the Calibre interface for everything except when things appear to go wrong when I use the normal Widows tools to investigate, not to fix.
For the sake of even further clarity I am using Windows 7 32-bit, fully up to date, 4 GB RAM, and the data is in a RAID mirrored set. 35 years in IT means that while I am no less likely to cock things up than anyone else, I would normally expect to realise that I had done something stupid
Thanks for all the responses, I hope that this is not coming across as negative because I don't intend it to be, it's just that Calibre is so close to perfect for me that I want to be confident in it, and after I have got over the initial hurdles would like to get involved in the development.

[Adoby]

I add and remove books and edit meta-data all the time. No problem.

Can you repeat the bug? And give simple instructions so others can repeat it?

[DoctorOhh]

Originally Posted by paulrw

To clarify things more, I am using the Calibre interface for everything except when things appear to go wrong when I use the normal Widows tools to investigate, not to fix.

That clarifies my questions. It sounds like you're doing everything correct.

Originally Posted by paulrw

Thanks for all the responses, I hope that this is not coming across as negative because I don't intend it to be, it's just that Calibre is so close to perfect for me that I want to be confident in it, and after I have got over the initial hurdles would like to get involved in the development.

It doesn't sound negative at all. You're just getting the chance to experience the joy of frustration problems we haven't experienced ourselves. Hopefully you'll figure it out before the rest of us are afflicted with the same good fortune calamity that has fallen on you.

At this point Kovid tells the person to start calibre from the command prompt calibre-debug -g and capture the data in the command window.

[kovidgoyal]

You've likely got a corrupted install. None of the behavior you're describing is normal, including the error message which indicates that one of the files in the calibre install is corrupted.

And all this would lead me to believe you are having disk problems. I dont know if newer versions of windows come with disk checking tools, but if they do, I suggest running a disk check.

Also uninstall and re-install. And make sure you dont have any calibre files lying around after the uninstall, which might happen if the files checksums have changed thanks to disk corruption.

[theducks]

Originally Posted by paulrw

For the sake of even further clarity I am using Windows 7 32-bit, fully up to date, 4 GB RAM, and the data is in a RAID mirrored set. 35 years in IT means that while I am no less likely to cock things up than anyone else, I would normally expect to realise that I had done something stupid

Is that a Local RAID or a Network RAID?
AFAICT Calibre writes the DB and then the files. Cached writes could really farkle things up

[paulrw]

Is that a Local RAID or a Network RAID?

Local.