Poor performance with large library

[BartZorn]

Originally Posted by BetterRed

That database size, 61MB for 30,000 books, doesn't seem abnormal to me, others may care to comment.

Could you run calibre at the server in debug mode and post the log here?

That it is so slow when run from server suggests the server itself is the bottleneck, rather that anything related to the network.

What file system is the server running (NTFS or ??), how much memory does it have, and what other applications run at the server?

Have you tried relocating the database, as suggested in posts 3 and 4.

BR

OK. My server is a Core i5-4570, with 16 GB memory, running Windows Server 2016 standard edition. Two cores are occupied running a Hyper-V VM doing video conversions. The other two are doing only what a lightly loaded Windows Server does.

I did "calibre-debug -g >calibre.log" and lo and behold, startup took about 17 seconds. The log file is attached to this post.
Next I tried a normal startup of calibre, and that also took less than 20 seconds.

The library and the database are all on a local disk on this server, disk G:, not C:.

It seems that the problem has disappeared by itself. Not very satisfying.

Thanks for all your suggestions!

Bart


Bart

[BartZorn]

The forum site seems to think that a .log file is an invalid file. Strange. But it is so small, here it is:

calibre 3.7 embedded-python: True is64bit: False
Windows-10-10.0.14393-SP0 Windows ('32bit', 'WindowsPE')
32bit process running on 64bit windows
('Windows', '10', '10.0.14393')
Python 2.7.12+
Windows: ('10', '10.0.14393', 'SP0', u'Multiprocessor Free')
Interface language: None
Turning on automatic hidpi scaling
devicePixelRatio: 1.0
logicalDpi: 96.0 x 96.0
physicalDpi: 141.76744186 x 141.402061856
Using calibre Qt style: True
Starting up...
Started up in 17.28 seconds with 36266 books

That's all

Bart

[Fiat_Lux]

Originally Posted by BartZorn

Previously, the size of the database did not seem to be important, but now I have to wait 5 minutes or more for a single change.

a) What extensions have you installed;
b) How much RAM does your system have;
c) When Calibre is running, how much free RAM is available;

OK, we have a ridiculous large database, over 30000 entries.

Large is when you have 100,000 entries in your Calibre library.
Huge is when you have 1,000,000 entries in your Calibre library.

Amber

[BartZorn]

Originally Posted by Fiat_Lux

a) What extensions have you installed;
b) How much RAM does your system have;
c) When Calibre is running, how much free RAM is available;



Large is when you have 100,000 entries in your Calibre library.
Huge is when you have 1,000,000 entries in your Calibre library.

Amber

a) none
b) 16 GB, but about 4 GB is occupied by a VM.
c) about 6 GB

OK, I stand corrected, my library is not big.

Bart

[BetterRed]

Originally Posted by BartZorn

OK. My server is a Core i5-4570, with 16 GB memory, running Windows Server 2016 standard edition. Two cores are occupied running a Hyper-V VM doing video conversions. The other two are doing only what a lightly loaded Windows Server does.

I did "calibre-debug -g >calibre.log" and lo and behold, startup took about 17 seconds. The log file is attached to this post.
Next I tried a normal startup of calibre, and that also took less than 20 seconds.

The library and the database are all on a local disk on this server, disk G:, not C:.

It seems that the problem has disappeared by itself. Not very satisfying.

Nah - something must have changed, I suspect you know that - Windows Yipdate, AntiViral... etc

I don't believe in fairies, or hobgoblins - unless they're of the kind mentioned in Menken's "In Defense of Women". But let's hope the cure sticks - whatever it was.

Were any video conversions running - I know they run in a VM but...

Originally Posted by BartZorn

calibre 3.7 embedded-python: True is64bit: False
Windows-10-10.0.14393-SP0 Windows ('32bit', 'WindowsPE')
32bit process running on 64bit windows
('Windows', '10', '10.0.14393')
Python 2.7.12+
Windows: ('10', '10.0.14393', 'SP0', u'Multiprocessor Free')
Interface language: None
Turning on automatic hidpi scaling
devicePixelRatio: 1.0
logicalDpi: 96.0 x 96.0
physicalDpi: 141.76744186 x 141.402061856
Using calibre Qt style: True
Starting up...
Started up in 17.28 seconds with 36266 books

Looks like you're running the the 32bit version of calibre. Any specific reason for that, not saying you shouldn't, or that it is the cause of the poor performance, but why use the Windows on Windows detour when there's no need.

BR

[theducks]

Originally Posted by BetterRed

That database size, 61MB for 30,000 books, doesn't seem abnormal to me, others may care to comment.

Could you run calibre at the server in debug mode and post the log here?

That it is so slow when run from server suggests the server itself is the bottleneck, rather that anything related to the network.

What file system is the server running (NTFS or ??), how much memory does it have, and what other applications run at the server?

Have you tried relocating the database, as suggested in posts 3 and 4.

BR

38MB here for less than 1/2 that many books. That's a thin DB

[theducks]

Originally Posted by BartZorn

The forum site seems to think that a .log file is an invalid file. Strange. But it is so small, here it is:

calibre 3.7 embedded-python: True is64bit: False
Windows-10-10.0.14393-SP0 Windows ('32bit', 'WindowsPE')
32bit process running on 64bit windows
('Windows', '10', '10.0.14393')
Python 2.7.12+
Windows: ('10', '10.0.14393', 'SP0', u'Multiprocessor Free')
Interface language: None
Turning on automatic hidpi scaling
devicePixelRatio: 1.0
logicalDpi: 96.0 x 96.0
physicalDpi: 141.76744186 x 141.402061856
Using calibre Qt style: True
Starting up...
Started up in 17.28 seconds with 36266 books

That's all

Bart

Why are you running the 32 bit version?

i5 don't make the best ' file Server' machines: Xeon is more common in server room racks

[sydmalicious]

Originally Posted by Fiat_Lux

Large is when you have 100,000 entries in your Calibre library.

Huge is when you have 1,000,000 entries in your Calibre library.



Amber

And I thought I was an ebook hoarder with 4,000 books...😏

Where do people get this many books? How much money must One spend for a MILLION BOOK library? 😜


Sent from my iPad using Tapatalk Pro

[BartZorn]

Originally Posted by theducks

Why are you running the 32 bit version?

i5 don't make the best ' file Server' machines: Xeon is more common in server room racks

I did not realize that a 64 bit version exists. Where can I download it? The normal download page does not give a hint.

Bart

[itimpi]

Originally Posted by BartZorn

I did not realize that a 64 bit version exists. Where can I download it? The normal download page does not give a hint.

Bart

The normal download page has an option labelled Windows (64-bit)! If you are not seeing that you are downloading from somewhere else.

[BetterRed]

Originally Posted by BartZorn

I did not realize that a 64 bit version exists. Where can I download it? The normal download page does not give a hint.

Bart

Eh, where have you been downloading from



BR

[BartZorn]

Yes, sorry. I have always downloaded newer versions after being prompted by the running version, 32 bit.

I will replace it by the 64 bit version.

Thanks,

Bart

[JSWolf]

The 64-bit version should be faster as it will use more memory and have less swapping.

[Divingduck]

@BartZorn
When the performance issue happen again check with the task manager and resource manager your system performance. This will maybe give you additional hints for improvements.

[BartZorn]

I now have the 64 bit version installed. Difference is hardly noticeable.

Bart