Slow news downloading?

[drwowe]

I thing I recall a preference in older version of Calibre where you could specify the maximum number of news sources to run in parallel, and it would download them all in parallel.

With the current 0.6.2x series it only seems to work on one feed at a time, and if that one feed is slow then everybody else has to wait.

Can we get back the multiple feeds in parallel feature?

[kovidgoyal]

There's no such limitation in newer calibre. The number of parallel jobs is the same as the number of processor cores on your machine. See Preferences->Advanced.

[drwowe]

Originally Posted by kovidgoyal

There's no such limitation in newer calibre. The number of parallel jobs is the same as the number of processor cores on your machine. See Preferences->Advanced.

It seems to ignore that preference. Can I make the number higher than the number of cores? The limiting factor for me isn't CPU time.

[kovidgoyal]

No it's upper limit is the number of cores

[drwowe]

What the purpose of this limit? It seems kind of arbitrary.

I removed the line "limit = min(limit, cpu_count())" from server.py and my news downloading is much faster now because the entire queue doesn't block when once news source stalls.

Even with 10 parallel downloads the average CPU load is under 5% because most time is spent waiting for server responses.

[kovidgoyal]

Because without that limit, people complain that calibre causes their computers to run slow/overheat/blah blah.

[drwowe]

Originally Posted by kovidgoyal

Because without that limit, people complain that calibre causes their computers to run slow/overheat/blah blah.

Why not set the limit to the number of cores during first setup?
After that, anyone playing with "Advanced" preferences has no right to complain...

edit:
or if that still makes it too easy for users to shoot themselves, how about reading a variable ncpus from global.py (with no GUI). If it is set, then it overrides the detected number. Then nobody's computer overheats unless they Edit A Configuration File!

[kovidgoyal]

Because on modern machines that would mean 8-16 waiting processes and then people would complain about memory consumption, as if there is any fundamental difference between used and unused memory

But open a ticket and I'll add a checkbox to disable the cpu_count check.

[drwowe]

Originally Posted by kovidgoyal

Because on modern machines that would mean 8-16 waiting processes and then people would complain about memory consumption, as if there is any fundamental difference between used and unused memory

But open a ticket and I'll add a checkbox to disable the cpu_count check.

Done. Ticket #4213. I added a patch but you can ignore it if you want to do it differently.

[kovidgoyal]

The option is in Preferences->Misc