Unable to import large db keep getting an error

[Student1]

Hi,

i'm using the latest release and seem to keep getting an error while importing a large amount of books. Using windows vista 64 bit, this used to work fine before but lately i keep getting the same problem.

here is the error message :




any idea what could be causing this? Tried to uninstall and reinstall and i keep getting this error each time i try to import some 10000 books.

any help is appreciated thanks!

[kovidgoyal]

Odd, nothing has changed in the add books code in the latest release. I suggest turning off you Antivirus while doing the import

[Student1]

Yes its the first thing i tried. I just uninstalled caliber now again, cleaned all the option, deleted db and starting from scratch... crossing fingers !!

[Student1]

Also i noticed that when you remove the check for "read metadata", so it creates metadata from filename, it still reads it when it tries to import the books, sometimes for some weird reason it even gets stuck on some books (then some other times it has no problem with them).

[kovidgoyal]

I think that option is broken. If you want to disable reading of metadata from files use the plugins part of the config dialog and disable the metadata reading plugins.

[kovidgoyal]

Actually I take that back, that option is not broken. Note that even with it unchecked the add books dialog will still say reading metadata, only now it will read from the filenames

[Student1]

Ok thanks that explains the speed up ! Still for some reason it hangs on certain files while reading the meta information.

[kovidgoyal]

Run it from a terminal as

calibre-debug -g

and post any output you see

[Student1]

tried it on a another computer and got the same problem, i'm doing it right now from a command line, i ll let you know what i get!

edit : I think i found the problem, i stoped the process yesterday before the temp files were deleted. I have now deleted the temp files the software created and it seems to be working. Its at 22% and no error yet. The console is opened and monitoring it, so if something happens i ll let you know! Thanks for the help!

[Student1]

Works just fine now. note to self : If i ever stop a process make sure to delete the leftover temp files ;p!

[Student1]

(( After 20 so hours it failed here is the console info :

C:\Program Files (x86)\calibre>calibre -debug -g

C:\Program Files (x86)\calibre>calibre-debug -g
link hasn't been detected!
link hasn't been detected!
Building file list...
Parsing Science fiction & Fantasy\Karl Edgar Wagner\Karl Edward Wagner -
Misericorde.html
Open ebook created in c:\users\maledi~1\appdata\local\temp\calibre_0.4.1 39_ljzja
n_create_oebzip
Output saved to c:\users\maledi~1\appdata\local\temp\calibre_0.4.1 39_vxgpie_plug
in_html2zip.zip
Building file list...
Parsing Science fiction & Fantasy\Karl Edgar Wagner\Karl Edward Wagner -
At First Just Ghostly.html
Open ebook created in c:\users\maledi~1\appdata\local\temp\calibre_0.4.1 39_57twz
p_create_oebzip
Output saved to c:\users\maledi~1\appdata\local\temp\calibre_0.4.1 39_pyzwpg_plug
in_html2zip.zip
Traceback (most recent call last):
File "calibre\gui2\add.pyo", line 205, in run
File "calibre\library\database2.pyo", line 1216, in import_book
File "calibre\library\database2.pyo", line 865, in set_metadata
File "calibre\library\database2.pyo", line 934, in set_title
File "calibre\library\sqlite.pyo", line 146, in run
DatabaseException: unable to open database file
RemoteTraceback (most recent call last):
File "calibre\library\sqlite.pyo", line 119, in run
OperationalError: unable to open database file

QObject::setParent: Cannot set parent, new parent is in a different thread
QPixmap: It is not safe to use pixmaps outside the GUI thread
QPixmap: It is not safe to use pixmaps outside the GUI thread
QPixmap: It is not safe to use pixmaps outside the GUI thread
QPixmap: It is not safe to use pixmaps outside the GUI thread
QPixmap: It is not safe to use pixmaps outside the GUI thread
QObject::startTimer: timers cannot be started from another thread
QPixmap: It is not safe to use pixmaps outside the GUI thread
QPixmap: It is not safe to use pixmaps outside the GUI thread

Was trying simply to import 13000 volumes... and it failed. Does the error message give you any indication why it failed?

thanks!

[kovidgoyal]

Hmm that output is not helpful. Basically its saying that it's unable to access the database, and I have no idea why that would happen. I suggest breaking up the import into smaller chunks.

[Student1]

Thanks, thats what i started doing. Even if it failed it was able to import 12000 volumes just fine. And because it does it in alphabetical order i m just adding the missing ones. Atleast we have a work arround thanks for your troubles!

[Student1]

Well i finally was able to import 17000 books. I cut the upload to db in 4 ups but now i m trying to add a few more and reading recursively giving me some serious headaches. It starts trying to read and randomly stops at different places. First time i imported the db i was kind of lucky and it read and imported everything (had to try 10 times), but now importing just a few books is almost impossible with the meta read. seems to get stock all the time. Any idea what could be causing the read to randomly stop, debug doesn t even say anything... just stops . (each time at a different spot).

Edit: I was finally able to read the dir i was trying to import, but took 7 times. Now its importing, but this has to be some sort of issue with something, memory leak or something random... weird.

[kovidgoyal]

Hmm you're saying that you can't add even a fewmore books after your database has reached 17k entries?