bulk operations - a couple of questions

[ms233]

Hi there! I'm brand new here, so hopefully I'm posting this in the right place.

I'm using Calibre (huge love!) to manage a very large collection (100,000 books, about 100 GB) and I've run into a couple of problem areas.

First, with bulk cover and metadata downloads, I run into a python error when I try to update 4000 books at a time. If I break it up and do batches of say 1000 at a time, it seems ok. Is there any way to overcome this limitation? I have been processing in batches like this, but it would be way less labor intensive to do them all at once.

Second issue is copying from one library to another seems to slow down dramatically as the job progresses. The first few books copy really quickly, but as time goes on, I notice it taking longer and longer to process each book. Copying 4000 books takes many hours. Is that normal, and is there any way to speed it up?

[theducks]

Hitting cover or metadata servers with huge number of requests is A NOT NICE practice to do to someone who provides us with free access to their metadata.

We have been blocked before for overwhelming servers that were simply intended for human users.

IMHO limit requests (20-30 at a time), there are millions of other Calibre users that need access to that metadata

[ms233]

I appreciate your input. I had no idea that this kind of thing was considered bad practice. I'd appreciate any links where I can read about it further.

In any case, back to the point at hand, my issues are more with the Calibre software itself. I was hoping to understand better how Calibre is behaving during these bulk operations, particularly the bulk library-to-library copy where the performance is degrading over time. Do you have any insights that might help me speed things up?

[BetterRed]

What OS are you using? What are your h/w specs?

You might want to consider putting the databases onto a PcIE SSD drive, and the library/author/book folders onto SATA 3.0

BR

[ms233]

Originally Posted by BetterRed

What OS are you using? What are your h/w specs?

You might want to consider putting the databases onto a PcIE SSD drive, and the library/author/book folders onto SATA 3.0

BR

I'm pursuing some hardware optimizations to try to improve performance, but I guess I was more hoping to catch a developer's attention to try to understand why the speed of the copy operation would degrade with each successive operation. The first copies take probably less than half a second per book. As time goes on, each book can take upwards to 5-10 seconds to process.

[kovidgoyal]

Because every copy has to check for duplicates against all existing books. As you copy more books the list of books to check against becomes longer.

[ms233]

Originally Posted by kovidgoyal

Because every copy has to check for duplicates against all existing books. As you copy more books the list of books to check against becomes longer.

Hi Kovid!

From my observation, each copy job starts fast and slows down as the job progresses. When I start a second job, the next job starts out fast and slows down. If it was the dupe checking, I'd think each job would get progressively slower than the one before it, which doesn't seem to be the case. For every job, the last 5% takes 10x longer than the first 5%. This is when copying a library of about 4000 titles into one that has about 100,000. Hopefully that makes sense.

Any insight on the python error when trying to download metadata for 4000+ titles? I'm guessing I'm hitting an maximum within the software for a list of books to update.

[kovidgoyal]

It would help if you posted the actual error message.And the suplicate check has to check not just the books in the library but also any previous dupes from the current job.

[rylee]

i have a similar issue. calibre will max out my cpu (I7-3930K) and memory (32GB). the last effort i killed it when memory was at 99%. the transfer gets to 78% then stops. i have to use task manager to kill the process because trying to use the abort button does nothing. i am thinking that perhaps the size i am trying to add (254GB) may be overwhelming the program.

[ms233]

Originally Posted by kovidgoyal

It would help if you posted the actual error message.And the suplicate check has to check not just the books in the library but also any previous dupes from the current job.

That makes a lot of sense. Thanks. I don't suppose there's any way to turn off the checking of duplicates from the current job?

If I manage to generate the python error again, I'll start a new thread and post it there. Thanks again!

[theducks]

Originally Posted by ms233

That makes a lot of sense. Thanks. I don't suppose there's any way to turn off the checking of duplicates from the current job?

If I manage to generate the python error again, I'll start a new thread and post it there. Thanks again!

Preferences: Adding books: (the various rules too)

[kovidgoyal]

Not for a currently running job, no.

[ms233]

Originally Posted by theducks

Preferences: Adding books: (the various rules too)

This has made a massive difference in the speed of mass imports. Thank you both very much for the help on this!

[pjmpjm]

Originally Posted by theducks

Hitting cover or metadata servers with huge number of requests is A NOT NICE practice to do to someone who provides us with free access to their metadata.

We have been blocked before for overwhelming servers that were simply intended for human users.

IMHO limit requests (20-30 at a time), there are millions of other Calibre users that need access to that metadata

Whoa! Had absolutely no idea about this. (Not that I make more than 6-8 requests at at time.) Good to know. Sometimes these things aren't obvious to the newbies.