Calibre hangs when importing books

[Cheepnis]

I'm trying to import a large number of books (.htm files) but unless I do only a few files at once, it seems to hang and calibre must be restarted. The files that were imported up to the fail seem to import, but I'm trying to import 10,000 books and can't do it a dozen at a time. Any help?

[Manichean]

Details on "seems to hang", please. Adding 10000 books at once will take quite some time, how exactly did you determine Calibre to be frozen?
Depending on the types of books you're importing there may be problems reading the metadata. You could try disabling the PDF metadata reading plugin before importing and see if that changes anything.

[Cheepnis]

Seems to hang = "Not Responding" error on the window title bar and no advance in the progress bar or title names being imported. Task manager shows no cpu activity from calibre. The only recovery option is to close calibre and restart it again. I've even tried leaving it overnight to see if it "catches up" but it's truly frozen.

It doesn't seem to hang at the same point or on the same file. It will hang at different times - sometimes after adding (for example) 1200 books, sometimes after only 120.

I'm not trying to add 10,000 at one time. I first tried only 2500, then 1000, then 250 at a time. Even with 100, it still might hang (although, the smaller the volume, the more likely it would succeed).

This would all be tolerable, because I could just restart in the file list wherever it crashed except for one thing: The filenames are all just a series of letters, so I don't know where it ended. Furthermore, I can't simply use a file count (300 books succeed, so resume at the 301st file) and here's why: They're all htm files and calibre gets the book title from the html header info (<title></title>) instead of the unique and correct <h1></h1> location where I would like calibre to look. Unfortunately, when these files were originally made, they occasionally used the same <title></title> header info from other books, so caliber sees them as duplicates. It will then never import those because when it hangs, it never asks if I want to include duplicates and thus leaves them out. So my file count isn't reflective of where it left off.

If I could get calibre to use <h1></h1> as the title source (and <h3></h3> for author which it also never sees), then I could better rely on a file count to resume after the hang, as there shouldn't be any duplicates. I've only see a way to add or change expressions for getting data from the filename, but no way to tweak the way it gets data from the file contents.

Sorry if this is all noob stuff and I appreciate your help!

[Manichean]

OK, so Calibre seems to genuinely have hung itself- I was asking because people often assume that operations taking longer than they anticipated have frozen, which isn't the case most of the time.

I don't think you can customize the internal metadata reading short of editing the source code, but in the add books- preferences page, you should be able to select what happens when Calibre believes it has found a duplicate. One of those options should allow you to add the book files anyway.

The accepted method seems to be to add larger batches of files in smaller chunks of, say, some 2000 files or so. Such a chunk tends to get added all in one. All considered, that shouldn't be too difficult with your situation and would probably be the easiest way.

EDIT: I just realized I overlooked that you already did smaller chunks. Disregard the last paragraph, then.

[DoctorOhh]

Originally Posted by Cheepnis

This would all be tolerable, because I could just restart in the file list wherever it crashed except for one thing: The filenames are all just a series of letters, so I don't know where it ended.

I have no idea why it is hanging. But you can get a better idea of what files completed by leaving the job details window open from the start.

Good Luck.

[kovidgoyal]

I suggest you run the import from the command line, better control and less likelihood of crashes.

calibredb add -h

in a command prompt.

[farrun100]

I am a great fan of Calibre and acquaint a lot of my clients and friends about it.

However....

I too am experiencing hangs while trying to load new books. My needs are more modest, 50 books but I have had these hangs with 10 or less in the load list. Would it be possible to create a job per added book and provide a way to cancel the job if it hangs? Can there be some details added to the job that might help in the debugging of this situation? Right now if I click the progress bar, it says "no details available" but I am at 29% of the load. I have suspected that some books may cause errors but it is difficult to test this because there is no easy way to determine what book it is hung on.

I looked at the suggestion of using the command line and have to ask if "file" can be a folder of ebooks with 1 book per subfolder otherwise trying to list a large number of files would be very time consuming.

I am using 0.8.35 and my reader is the Aluratek Libre.


Thanks for a great product.

[DoctorOhh]

Originally Posted by farrun100

Right now if I click the progress bar, it says "no details available" but I am at 29% of the load. I have suspected that some books may cause errors but it is difficult to test this because there is no easy way to determine what book it is hung on.

You say the same issue, but clearly this is a similar issue but not the same issue. The OP stated that calibre hung with 0% cpu load. Clearly this is hung. You state calibre hung with 29% cpu load, typically this means calibre is in fact still working.

Originally Posted by farrun100

I looked at the suggestion of using the command line and have to ask if "file" can be a folder of ebooks with 1 book per subfolder otherwise trying to list a large number of files would be very time consuming.

The most common problem when importing books is the PDF metadata reader crashing. Try Manichean's advice in post 2 and disable this plugin (Preferences - Plugins - Metadata Reader plugins.)

I've never run it from the command line so can't help you there.

Good Luck.

[xxrachxx]

I am having a similar problem, however when i am trying to import books to calibre, even if it is just one book, it hangs at 0% and doesn't move no matter how long i leave it for.

[speakingtohe]

@ xxrachxx
If you have already tried Walt's advice, try running a database check, check disk space, make sure books are not read only, ie from a DVD or CD.
Helen