Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre

Notices

Reply
 
Thread Tools Search this Thread
Old 12-12-2009, 02:02 AM   #1
drwowe
Enthusiast
drwowe began at the beginning.
 
Posts: 27
Karma: 10
Join Date: Apr 2008
Device: none
Slow news downloading?

I thing I recall a preference in older version of Calibre where you could specify the maximum number of news sources to run in parallel, and it would download them all in parallel.

With the current 0.6.2x series it only seems to work on one feed at a time, and if that one feed is slow then everybody else has to wait.

Can we get back the multiple feeds in parallel feature?
drwowe is offline   Reply With Quote
Old 12-12-2009, 06:29 AM   #2
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 25,407
Karma: 4961459
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
There's no such limitation in newer calibre. The number of parallel jobs is the same as the number of processor cores on your machine. See Preferences->Advanced.
kovidgoyal is offline   Reply With Quote
Old 12-12-2009, 11:02 PM   #3
drwowe
Enthusiast
drwowe began at the beginning.
 
Posts: 27
Karma: 10
Join Date: Apr 2008
Device: none
Quote:
Originally Posted by kovidgoyal View Post
There's no such limitation in newer calibre. The number of parallel jobs is the same as the number of processor cores on your machine. See Preferences->Advanced.
It seems to ignore that preference. Can I make the number higher than the number of cores? The limiting factor for me isn't CPU time.
drwowe is offline   Reply With Quote
Old 12-12-2009, 11:34 PM   #4
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 25,407
Karma: 4961459
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
No it's upper limit is the number of cores
kovidgoyal is offline   Reply With Quote
Old 12-14-2009, 11:36 AM   #5
drwowe
Enthusiast
drwowe began at the beginning.
 
Posts: 27
Karma: 10
Join Date: Apr 2008
Device: none
What the purpose of this limit? It seems kind of arbitrary.

I removed the line "limit = min(limit, cpu_count())" from server.py and my news downloading is much faster now because the entire queue doesn't block when once news source stalls.

Even with 10 parallel downloads the average CPU load is under 5% because most time is spent waiting for server responses.

Last edited by drwowe; 12-14-2009 at 11:38 AM.
drwowe is offline   Reply With Quote
Old 12-14-2009, 12:03 PM   #6
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 25,407
Karma: 4961459
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Because without that limit, people complain that calibre causes their computers to run slow/overheat/blah blah.
kovidgoyal is offline   Reply With Quote
Old 12-14-2009, 02:41 PM   #7
drwowe
Enthusiast
drwowe began at the beginning.
 
Posts: 27
Karma: 10
Join Date: Apr 2008
Device: none
Quote:
Originally Posted by kovidgoyal View Post
Because without that limit, people complain that calibre causes their computers to run slow/overheat/blah blah.
Why not set the limit to the number of cores during first setup?
After that, anyone playing with "Advanced" preferences has no right to complain...

edit:
or if that still makes it too easy for users to shoot themselves, how about reading a variable ncpus from global.py (with no GUI). If it is set, then it overrides the detected number. Then nobody's computer overheats unless they Edit A Configuration File!

Last edited by drwowe; 12-14-2009 at 02:52 PM.
drwowe is offline   Reply With Quote
Old 12-14-2009, 03:18 PM   #8
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 25,407
Karma: 4961459
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Because on modern machines that would mean 8-16 waiting processes and then people would complain about memory consumption, as if there is any fundamental difference between used and unused memory

But open a ticket and I'll add a checkbox to disable the cpu_count check.
kovidgoyal is offline   Reply With Quote
Old 12-14-2009, 04:00 PM   #9
drwowe
Enthusiast
drwowe began at the beginning.
 
Posts: 27
Karma: 10
Join Date: Apr 2008
Device: none
Quote:
Originally Posted by kovidgoyal View Post
Because on modern machines that would mean 8-16 waiting processes and then people would complain about memory consumption, as if there is any fundamental difference between used and unused memory

But open a ticket and I'll add a checkbox to disable the cpu_count check.
Done. Ticket #4213. I added a patch but you can ignore it if you want to do it differently.
drwowe is offline   Reply With Quote
Old 06-23-2011, 12:11 PM   #10
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 25,407
Karma: 4961459
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
The option is in Preferences->Misc
kovidgoyal is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Calibre not downloading scheduled news rb9999 Calibre 15 02-23-2011 04:20 PM
Problems downloading news Sridhar Calibre 3 08-29-2010 02:49 PM
News source downloads always slow ckole Calibre 7 01-23-2010 07:48 AM
Problem downloading news (Ubuntu 8.10) Kaaahn Calibre 1 04-12-2009 12:00 PM


All times are GMT -4. The time now is 11:41 AM.


MobileRead.com is a privately owned, operated and funded community.