MobileRead Forums
Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre
Reload this Page Slow news downloading?

Welcome to the MobileRead Forums.

You are currently viewing our boards as a guest which gives you limited access to view most discussions and access our other features. By joining our free community today, you will have fewer ads, access to post topics, communicate privately with other members, respond to polls, upload content and access many other special features.

If you have any problems with the registration process or your account login, please contact us.

Hint: Don't have time to visit us daily? Subscribe to our main RSS feed to receive our frontpage posts at your convenience.

Notices

Calibre Calibre is an open-source library manager to view, convert and catalog e-books. Cross-platform (Linux, Windows and OS X)

Reply
 
Thread Tools Search this Thread Display Modes
Old 12-12-2009, 02:02 AM   #1
drwowe
Member
drwowe began at the beginning.
 
Posts: 21
Karma: 10
Join Date: Apr 2008
Device: none
Slow news downloading?
I thing I recall a preference in older version of Calibre where you could specify the maximum number of news sources to run in parallel, and it would download them all in parallel.

With the current 0.6.2x series it only seems to work on one feed at a time, and if that one feed is slow then everybody else has to wait.

Can we get back the multiple feeds in parallel feature?
drwowe is offline   Reply With Quote
 
Enthusiast
Old 12-12-2009, 06:29 AM   #2
kovidgoyal
Creator of calibre, Ph.D.
kovidgoyal is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!kovidgoyal is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!kovidgoyal is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!kovidgoyal is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!kovidgoyal is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!kovidgoyal is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!kovidgoyal is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!kovidgoyal is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!kovidgoyal is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!kovidgoyal is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!kovidgoyal is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!
 
kovidgoyal's Avatar
 
Posts: 12,367
Karma: 50287
Join Date: Oct 2006
Location: Albuquerque, NM
Device: PRS-500/505/600/700, K2, BeBook, ShineBook, Nook
There's no such limitation in newer calibre. The number of parallel jobs is the same as the number of processor cores on your machine. See Preferences->Advanced.
__________________
Get calibre

Notice to all: I can not provide assistance with DRM removal, for legal reasons, so please do not contact me about it.
kovidgoyal is online now   Reply With Quote
Old 12-12-2009, 11:02 PM   #3
drwowe
Member
drwowe began at the beginning.
 
Posts: 21
Karma: 10
Join Date: Apr 2008
Device: none
Quote:
Originally Posted by kovidgoyal View Post
There's no such limitation in newer calibre. The number of parallel jobs is the same as the number of processor cores on your machine. See Preferences->Advanced.
It seems to ignore that preference. Can I make the number higher than the number of cores? The limiting factor for me isn't CPU time.
drwowe is offline   Reply With Quote
Old 12-12-2009, 11:34 PM   #4
kovidgoyal
Creator of calibre, Ph.D.
kovidgoyal is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!kovidgoyal is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!kovidgoyal is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!kovidgoyal is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!kovidgoyal is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!kovidgoyal is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!kovidgoyal is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!kovidgoyal is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!kovidgoyal is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!kovidgoyal is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!kovidgoyal is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!
 
kovidgoyal's Avatar
 
Posts: 12,367
Karma: 50287
Join Date: Oct 2006
Location: Albuquerque, NM
Device: PRS-500/505/600/700, K2, BeBook, ShineBook, Nook
No it's upper limit is the number of cores
__________________
Get calibre

Notice to all: I can not provide assistance with DRM removal, for legal reasons, so please do not contact me about it.
kovidgoyal is online now   Reply With Quote
Old 12-14-2009, 11:36 AM   #5
drwowe
Member
drwowe began at the beginning.
 
Posts: 21
Karma: 10
Join Date: Apr 2008
Device: none
What the purpose of this limit? It seems kind of arbitrary.

I removed the line "limit = min(limit, cpu_count())" from server.py and my news downloading is much faster now because the entire queue doesn't block when once news source stalls.

Even with 10 parallel downloads the average CPU load is under 5% because most time is spent waiting for server responses.
Last edited by drwowe; 12-14-2009 at 11:38 AM.
drwowe is offline   Reply With Quote
Old 12-14-2009, 12:03 PM   #6
kovidgoyal
Creator of calibre, Ph.D.
kovidgoyal is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!kovidgoyal is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!kovidgoyal is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!kovidgoyal is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!kovidgoyal is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!kovidgoyal is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!kovidgoyal is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!kovidgoyal is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!kovidgoyal is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!kovidgoyal is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!kovidgoyal is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!
 
kovidgoyal's Avatar
 
Posts: 12,367
Karma: 50287
Join Date: Oct 2006
Location: Albuquerque, NM
Device: PRS-500/505/600/700, K2, BeBook, ShineBook, Nook
Because without that limit, people complain that calibre causes their computers to run slow/overheat/blah blah.
__________________
Get calibre

Notice to all: I can not provide assistance with DRM removal, for legal reasons, so please do not contact me about it.
kovidgoyal is online now   Reply With Quote
Old 12-14-2009, 02:41 PM   #7
drwowe
Member
drwowe began at the beginning.
 
Posts: 21
Karma: 10
Join Date: Apr 2008
Device: none
Quote:
Originally Posted by kovidgoyal View Post
Because without that limit, people complain that calibre causes their computers to run slow/overheat/blah blah.
Why not set the limit to the number of cores during first setup?
After that, anyone playing with "Advanced" preferences has no right to complain...

edit:
or if that still makes it too easy for users to shoot themselves, how about reading a variable ncpus from global.py (with no GUI). If it is set, then it overrides the detected number. Then nobody's computer overheats unless they Edit A Configuration File!
Last edited by drwowe; 12-14-2009 at 02:52 PM.
drwowe is offline   Reply With Quote
Old 12-14-2009, 03:18 PM   #8
kovidgoyal
Creator of calibre, Ph.D.
kovidgoyal is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!kovidgoyal is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!kovidgoyal is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!kovidgoyal is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!kovidgoyal is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!kovidgoyal is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!kovidgoyal is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!kovidgoyal is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!kovidgoyal is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!kovidgoyal is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!kovidgoyal is faster than a rolling 'o,' stronger than silent 'e,' and leaps capital 'T' in a single bound!
 
kovidgoyal's Avatar
 
Posts: 12,367
Karma: 50287
Join Date: Oct 2006
Location: Albuquerque, NM
Device: PRS-500/505/600/700, K2, BeBook, ShineBook, Nook
Because on modern machines that would mean 8-16 waiting processes and then people would complain about memory consumption, as if there is any fundamental difference between used and unused memory

But open a ticket and I'll add a checkbox to disable the cpu_count check.
__________________
Get calibre

Notice to all: I can not provide assistance with DRM removal, for legal reasons, so please do not contact me about it.
kovidgoyal is online now   Reply With Quote
Old 12-14-2009, 04:00 PM   #9
drwowe
Member
drwowe began at the beginning.
 
Posts: 21
Karma: 10
Join Date: Apr 2008
Device: none
Quote:
Originally Posted by kovidgoyal View Post
Because on modern machines that would mean 8-16 waiting processes and then people would complain about memory consumption, as if there is any fundamental difference between used and unused memory

But open a ticket and I'll add a checkbox to disable the cpu_count check.
Done. Ticket #4213. I added a patch but you can ignore it if you want to do it differently.
drwowe is offline   Reply With Quote
Reply

« Previous Thread | Next Thread »
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes
Linear Mode Linear Mode

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Calibre not downloading scheduled news rb9999 Calibre 13 03-12-2010 10:40 AM
calibre - very slow conversion, very slow on PRS cremofix Calibre 3 06-10-2009 04:21 PM
Problem downloading news (Ubuntu 8.10) Kaaahn Calibre 1 04-12-2009 12:00 PM
AP News Begins Charging Web News Providers Bob Russell Lounge 1 04-20-2005 10:38 AM


All times are GMT -4. The time now is 06:20 PM.

Contact Us - FAQ - Guidelines - Privacy - Archive - Top

MobileRead.com is a privately owned, operated and funded community.