Looking for performance tweaks with the GUI and large amounts of ebooks

[seru1us]

Hey everyone!

It's been a while since I posted here, but essentially I really only use the Calibre GUI for sorting incoming eBooks, and it has begun to get to the point where it needs to be closed and re-launched a few times a day due to performance issues. Are there any tweaks I can do as far as disabling the caching of covers, or disable any unused features to speed it up a bit? Currently, the only things I am using the GUI for are:

- editing and downloading metadata
- transferring ebooks from one library to another
- running calibre-server (which should be its own dedicated process, but its on my to-do list)
- having a few sources running checks through calibredb to see if titles exist already (considering doing a pure db query instead of this for to speed things up)

And that's pretty much it. My current stats for the libraries are:

- "Production" Library with pristine, 99% perfect metadata: 78,392 eBooks
- "Staging" Library with little to no metadata to be manually confirmed: 40,578 eBooks
- "other" library with data that has probably no metadata anywhere online: 8,420 eBooks


Anyone have any suggestions? I already have a script that puts all of the sqlite databases and temp folders on a ramdisk prior to launching which helps a bit, but I'm still running into some performance issues. If you're curious about the script itself, here it is:

https://pastebin.com/wHYgczAa

Here is an example of the current memory usage of the application, which may or not be valuable:

https://i.imgur.com/AXUL8ac.png

[BetterRed]

Which OS, how much memory on the computer, which version of calibre?

Any columns built from other columns (composite columns) - they can slow things down, especially if they interrogate the file system itself.

Try running calibre with no plugins ==>> calibre

What does this -- "... having a few sources running checks through calibredb to see if titles exist already..." -- actually entail. I use the Find Duplicates plugin to do that - it's lightning fast because it uses the in-memory version of the database, and it can do exact, similar, soundex, and fuzzy matches, and x-library matches.

Because calibre loads the entire database into memory when a library is opened, the value of copying the database to a ram disk is debatable.

BR

[seru1us]

Originally Posted by BetterRed

Which OS, how much memory on the computer, which version of calibre?

Currently running Calibre 3.46 on Kubuntu 19.04. 64gigs of memory. My screenfetch

Originally Posted by BetterRed

Try running calibre with no plugins

I'll have to give this a shot to test some things, thanks. Currently the only plugins I'm running are:

Find Duplicates
Goodreads
Library Codes
Quality Check

Originally Posted by BetterRed

What does this -- "... having a few sources running checks through calibredb to see if titles exist already..." -- actually entail.

Yeah, I could have explained that better... So essentially I have an automated setup with FlexGet that checks to see if a book exists in my production calibre library (by only searching by title or isbn for performance reasons) *before* downloading the book. This way, it cuts down on a ton of internet traffic, storage space, and cleanup I have to to afterwards. So each time FlexGet finds a book to grab from one of my sources, I'll download it if it doesn't already exist.

Here is an example of that, don't judge the hard coded variables on github please

But having said that, Find Duplicates (i use this pretty liberally), Merge and Quality Check do a great job of picking up where this leaves off.

hope that helps

[BetterRed]

Is your FlexGet gadget loaded when you're running calibre. If it is, what happens if you shut it down.

Calibre searches are performed on the in-memory copy of the database. A search for a title in my ~150,000 item library is instantaneous - that's on a 16GB windows 10 machine. If I want to search in a library other than the one I have open in the GUI, I use the CalibreSpy plugin - its search times are likewise instantaneous.

In your first post you wrote "it has begun to get to the point where it needs to be closed and re-launched a few times a day due to performance issues.". In what respect are the 'performance issues' manifested - e.g. scrolling the book list, editing metadata ?

I don't, but I know of calibre users who run it 24x7 for days on end.

Pssst: other responders - I didn't look the OPs script or code. Maybe someone proficient at reading gobbledegook could take a peek. I only look at code when I want to have a sleep

BR

[kovidgoyal]

Since you report progressively degrading performance, that would imply a memory or other resource leak. Is the memory usage rising continuously? Does it happen with the server turned off?

[seru1us]

So just to confirm whether or not there is a memory leak possibility, I ran operations as normal for the past few days without closing it. Here are the results:

https://imgur.com/a/wODanQn

So you can see that the used memory has almost doubled with my normal activity. Also, just to clarify, I am probably running the server in a less than ideal fashion (from the GUI, not as a dedicated process).

If it is valuable, whenever I start the calibre process one of the things I do after mounting the ramdisk is piping the output of calibre-debug to a log, which I've uploaded to pastebin for this latest session:

https://pastebin.com/3SkKPC1U


I haven't tried this without the server running yet.... essentially because my main influx of ebooks into calibre comes from entries into the server. My workflow looks like this:

1. Flexget finds a book through an RSS feed, and does a calibre-db search to see if the title or isbn exists yet.
2. Flexget passes the info onto a separate program which grabs the book and adds it via calibre-db to my "eBooks_Staging" library
3. I use the GUI to download/confirm metadata, and then copy it into the "eBooks" library with all of my other books with verified metadata

One thing I've been thinking of is running the calibre-server as its own process, and then connecting to it from the gui instead of this lazy way I'm doing now. Hope all of this helps!

Details for BetterRed: I would say the initial slowdown is apparent when browsing the grid, and just trying to interact with the buttons. I don't think it can be nailed down to a specific function, all interactions are generally slowed.

[kovidgoyal]

First note that you cannot run the server and GUI via separate processes on the same computer. It has to be either in-process or on separate servers.

Some memory increase is to be expected, as the python memory allocator does not always release unused memory to the OS. A doubling from initial state is pretty normal. This happened over a few days? At a constant pace or mostly at the beginning?

[seru1us]

Originally Posted by kovidgoyal

First note that you cannot run the server and GUI via separate processes on the same computer. It has to be either in-process or on separate servers.

Some memory increase is to be expected, as the python memory allocator does not always release unused memory to the OS. A doubling from initial state is pretty normal. This happened over a few days? At a constant pace or mostly at the beginning?

Seems like it's been pretty consistent over the past couple of days. Based on the time between created logs, the memory usage expanded from the date of this thread's creation until today (approx 4 days).

And just to clarify, I would probably end up moving calibre-server to a container sometime in the future (and keep the GUI installed locally), since all of my other apps are running as containers now.

[kovidgoyal]

A continuous increase would indicate a mem leak. And in your debug log I see varios instances of worker processes being killed by the system. Is your system under memory pressure? You also appear to have some plugins installed, does th eleak happen if you run calibre with --ignore-plugins

[seru1us]

Update:

So I have been running calibre without "--ignore-plugins" and performing my normal tasks. Here is the result of the memory usage:

Start of Application - 1,128,176k RAM
Date: 8/22
https://i.imgur.com/iAR7ETI.png

Next Day - 5,772,288k RAM
Date: 8/23
https://i.imgur.com/iD2ZbCP.png

I didn't touch Calibre for a few days, but all of the automated calibredb processes were still running. So, no metadata downloading or editing, and it came out to this: 5,659,696k RAM
Date: 8/26
https://i.imgur.com/pWXlBhJ.png


So it seems as if the "leak" occurs only when downloading/editing metadata, and also seems consistent when running with "--ignore-plugins"

[kovidgoyal]

OK interesting, next step is to isolate wat operation actually causes the memory usage to increase. You should be able to note a significant increase after performing a handful of edits and/or downloads, see if you can narrow down what operation it is, then I can investigate.

[seru1us]

Originally Posted by kovidgoyal

OK interesting, next step is to isolate wat operation actually causes the memory usage to increase. You should be able to note a significant increase after performing a handful of edits and/or downloads, see if you can narrow down what operation it is, then I can investigate.

First things first, sorry to dig up an old thread. I just think I may have an update to this particular issue.

I am noticing a significant performance decrease and increase in memory usage when queries are hitting the calibre-server endpoint, and I am using the GUI. Just to reiterate my setup:

- Calibre GUI with server running
- Automated scripts using calibre-server to search/add books in separate library

The reason I wanted to update is because I disabled my scripts for a week or so and none of the performance issues arose. A few days ago I re-enabled them, and- lo and behold- calibre is using 20 gigs of ram. I have a suspicion that this is related to the using the calibre GUI and having external processes hit the server portion at the same time.

[kovidgoyal]

If you can come up with some minimal script that demonstrates the issue I will be happy to take a look.