library issue...

[lohtse]

I have a folder with 44000+ books in it and when I try to add the folder to calibre it gets to about 20% and calibre stops responding have left for several hours and still no response... have to close the application and then delete the calibre folder to continue..

Have tried creating a test folder with 1000 and 6000 books and same issue.. STOPS responding... And it is never on the same file either...

any ideas why this is happening folks?

[lohtse]

why move this thread to devices as it's an application issue?????

[theducks]

Originally Posted by lohtse

why move this thread to devices as it's an application issue?????

From the subforum title

Devices
Troubleshoot installation and connection issues specific to your device or OS, and learn how to make the most of calibre's device integration features

[speakingtohe]

Originally Posted by lohtse

why move this thread to devices as it's an application issue?????

How are you adding the books?

I usually drag and drop or use the 2nd or 3rd option from the add books menu.

Helen

[theducks]

Originally Posted by speakingtohe

How are you adding the books?

I usually drag and drop or use the 2nd or 3rd option from the add books menu.

Helen

44K books is a huge number to add all at once.

True Calibre does not have a stated 'at one time' limit, 44K books is more than most people have in their Library, let alone added all at one git-go.

The only time I did a big import was when I added all the titles (1 format) from the Baen CD's. Even that was done 1 CD at a time.

44K is a LOT of metadata to fix (and identify) if the books don't already contain excellent metadata (and covers)

20% implies 8K books were successfully imported before the OP deleted the attempt .
Breaking that 44K up into 5K chunks would probably get to the end quickest.

If the OP has a full backup of the source books, they could setup and use the AutoAdd folder. Moving a bunch of books to that folder, then move some more when the folder becomes empty

[speakingtohe]

Originally Posted by theducks

44K books is a huge number to add all at once.

True Calibre does not have a stated 'at one time' limit, 44K books is more than most people have in their Library, let alone added all at one git-go.

The only time I did a big import was when I added all the titles (1 format) from the Baen CD's. Even that was done 1 CD at a time.

44K is a LOT of metadata to fix (and identify) if the books don't already contain excellent metadata (and covers)

20% implies 8K books were successfully imported before the OP deleted the attempt .
Breaking that 44K up into 5K chunks would probably get to the end quickest.

If the OP has a full backup of the source books, they could setup and use the AutoAdd folder. Moving a bunch of books to that folder, then move some more when the folder becomes empty

e only tried adding 1000 books the second time. And it seemed to me that maybe he was making a separate folder and using auto add, although I don't know this.

Helen

[lohtse]

Originally Posted by speakingtohe

e only tried adding 1000 books the second time. And it seemed to me that maybe he was making a separate folder and using auto add, although I don't know this.

Helen

nope I move 1000 books to seperate folder and choose option 2 and later 3 in add books thing on calibre...always same result...


ps the auto add feature doesnt seem to work either!!!!

[Adoby]

Add 1K books at a time, or even less.

It is very likely that there are books that are damaged and so may cause the import to fail, and possibly hang calibre. You will have to try to figure out exactly what books are bad. And remove them and try again.

Also I believe that the process makes very heavy use of the temp folder. Make sure you have plenty of space on the drive that contains the temp folder.

The process is also very hard work for the computer. Verify that the harddrives are error free before you start and that there are no problems with the computer overheating.

[speakingtohe]

I'm pretty sure damaged books won't make it hang. I have accidentally imported movies, movies and other non book items. Still perhaps they do.

Helen

[theducks]

Originally Posted by speakingtohe

I'm pretty sure damaged books won't make it hang. I have accidentally imported movies, movies and other non book items. Still perhaps they do.

Helen

Movies are not 'Damaged files'

A corrupt Zip/EPUB/RAR (one that can't unzip) is a damaged file.

Calibre will accept incomplete books, as long as the container file is intact.

[speakingtohe]

Originally Posted by theducks

Movies are not 'Damaged files'

A corrupt Zip/EPUB/RAR (one that can't unzip) is a damaged file.

Calibre will accept incomplete books, as long as the container file is intact.

The only problem I have had is that the viewer won't open them, not that I have had that many damaged zips or epubs or pdf, but I have had some of each and so far they imported ok, just no covers and of course they are useless.

probably there is a special kind of damage that I have been lucky enough to avoid

Helen

[lohtse]

I think I will give in on calibre as have tried several times again using different files and amounts.... Always same result.. As to comments relating to temp folder and hard drives they are fine and plenty of space. Temp is fine and with 8gig of ram in a quadcore 64bit pc am sure the pc can cope too....

For me calibre is a pain in the Ar** lol

[speakingtohe]

Originally Posted by lohtse

I think I will give in on calibre as have tried several times again using different files and amounts.... Always same result.. As to comments relating to temp folder and hard drives they are fine and plenty of space. Temp is fine and with 8gig of ram in a quadcore 64bit pc am sure the pc can cope too....

For me calibre is a pain in the Ar** lol

Perhaps give it one shot with the drag and drop
Helen

[Adoby]

One way to handle this could be to use the command line tools instead. And add one book at a time in a script.

That way you can easily keep track of what books have been added and resume from the point where it failed. Most likely it would be a little slower, and require some scripting, but it would probably be much more reliable. And the script could run in the background in a terminal window while you sleep or do something else on the computer. (Not using Calibre.)

http://manual.calibre-ebook.com/cli/calibredb.html

If you have those 44K books in some sort of hierarchical folder structure, you could even add the path to the metadata when you add the book. That would most likely make it easier to add tags and other fields to make that library more manageable.

And good luck finding anything better than calibre...

[Sabardeyn]

Lohtse,
Where are the 44,000 ebooks that you want to add located?

It is possible that hardware configuration might be an issue. Calibre doesn't work well with network attached storage (NAS) or mapped drives, for instance. Some internet/cloud storage is problematic as well (Google Drive). And any HDD can have bad sectors or be failing.

Although less critical, have the source and destination drives been defragged recently? (Perhaps lots of drive head movement is causing delays which, added to other factors, is making things appear worse.)

I would also strongly urge adding books in much smaller groups. Start with 1 and then increase from there (5, 25, 50 , etc). Personally I would max out at adding 500 books or less at a time. I know you said you've tried several times and amounts, just not sure how small you went.

Unless you're using a "vanilla" install of calibre, it can be a bit difficult to get things arranged how you need/want them. The program's steep learning curve can also make things tough for new users. Once over those humps though, no other product comes anywhere close to calibre's functionality or feature set.