Performance issue with large library

[audeojude]

ok I changed the path to include the metadata.db and it worked.
so it would read like this

export CALIBRE_OVERRIDE_DATABASE_PATH="/home/scott/metadata.db"

I can tell that the database is being addressed and written to faster however the lag before the infterface is available is still around 20 seconds. Actually it feels just a hair longer.

I just added twenty books to the library and the dialog that shows them being added was much quicker. After it said they were added though, the time for the interface to become available was still about the same as before.

I then did a could of bulk metadata edits. One to change author name and another to add and then another to delete tags. These seemed to happen really quickly in the interface that shows progress compared to before. After that though the main interface took about 22 seconds to become responsive again.

So I think the database portion is running faster. However I am now going to say that I/O to it is not what was causing the unwieldy lag in the interface that I was complaining of.

I was thinking that maybe it was simply the interface trying to list 59000 books when it refreshes but I get the same lag even when there is only 41 entries listed on a search that has limited the return.

I am stumped.. I was so sure that what I was seeing was a I/O issue and it isn't. Moving the database to the ssd did improve I/O to it but that wasn't what was causing the percieved slowness.

If anyone has any ideas about what is causing this I'm open to theorys and stuff to try. I'm not sure how many people out there have that many files stored in calibre but I'm sure that over time more and more will have large libraries. It would be good to figure this out.

I will have to say though that calibre is amazing. Even though I have slow issues it still does the job and waiting 20 seconds isn't the end of the world. The only issue I have ever had with it that is a bug is that when adding a large number of files at one time it will sometimes crash. I just keep it down to smaller groups of books when adding to get around that. usually anything under 300 at a time will do fine. Though I have successfully at 600 and 700 a few time successfully.

I did notice that the ones that crashed felt more like it was the cumulative size of all the files rather than the number of them. Just an unsubstantiated feeling.

[kovidgoyal]

The biggest bottleneck is the Tag Browser. I cannot believe that hiding it makes no difference for you. For example, on my 20k test library with the tag browser hidden, clicking ok in the edit metadata dialog causes a one second delay before the screen is redrawn, with it open, the delay is five seconds.

Currently the Tag Browser implementation has to loop over all fields of all books to collect the statistical data, which is painfully slow. A database layer rewrite I am doing is going to eliminate most of that delay by switching to a normalized memory model instead of a per book one, but that rewrite is still along way from complete.

[audeojude]

I have tried it with it on and off.

ok here are some measured results..

edit metadata of one book with tag windo open.. 7 seconds from the time you hit the ok button till the main interface is available to use again. The update happened almost instantaneous to the database.

with the tag window closed doing the same edit it took only 1 second for the interface to be responsive again.

Here is where it gets weird. Since I most often am doing metadata edits on groups of books after I have added them I went ahead and chose 10 books and did the same test as above. Both with and without the tag window open.

With the tag window closed it took maybe 1.5 t o 2 seconds to write the data to the database but the program was unresponsive for 22 seconds.

With the tag window open the write to the database was the same but the window was unresponsive for 33 seconds.

so it does help to close the tag window but something about doing multiple metadata edits runs up the time before the interface is responsive. I don't notice a disparate amount of time between doing a metadata edit on 10 books vs 100 books other than a slightly longer database write at the front end where I can hear the drive running. I do notice a major difference between 1 book and 10

[audeojude]

ok tested with tag window open and adding a tag to 166 books
tag updated in 1.5 seconds.. 34.5 seconds till program became responsive again.

with tag window closed 1.5 seconds to remove the above added tag and 29 seconds for main program to become responsive again.

[audeojude]

kovidgoyal,

Just want to mention how much I love calibre. I have been looking for something like it for years. Found it about 6 months ago and it has made my digital life much much easier. Now my ideal would be for your calibre to integrate with readerware which I use for my physical library.

I suppose I could figure out how to import my readerware library as blank entries into calibre. I like readerware though because it allows me to use a barcode scanner on my books in bulk to import them as well as I have the network version and access it remotely from client computers. I have about 5000 books in that database and another 5000 or 6000 to go before it is fully populated. The hard part with that one is where to put 10 or 11 thousand physical books.


This reminds me.. I donated a few dollars after I started using calibre but I like to support products I use that are open source and as actively developed as yours is so I just kicked another few dollars your way I love it.
I hope that a lot of the other people that use it donate some here and there also.

[theducks]

Originally Posted by audeojude

kovidgoyal,


I suppose I could figure out how to import my readerware library as blank entries into calibre. I like readerware though because it allows me to use a barcode scanner on my books in bulk to import them as well as I have the network version and access it remotely from client computers.

Add from ISBN is what you are looking for .
I use this with a Keyboard Wedge scanner.
You can use it to add from Clipboard content (ISBN list).


Look at the CLI for creating Empty records in the DB (you will probably need to write a script)

[audeojude]

That would work for some of them to import them in but a huge number of the physical library are pre-ISBN.

I do see what you mean though about using a barcode scanner to populate a text document with ISBN codes and then importing those into calibre.

For me the goal is to have someone else do the programming and me just use the software. I don't have a particular bent or passion towards programming. I have passion for reading and collecting books. So for me its all about finding and using solutions others have made to meet my ends. that's one of the reasons I always donate towards the opensource projects of the software I use such as calibre, or egroupware and any of another 7 or 8 software packages I use a lot from open source developers. Much rather support those people and that model over most closed source developers. The products tend to be better, more bug free, and have better support from users online.

[kovidgoyal]

So you're finding there is a large lag after a bulk metadata edit but not after a single edit?

And thanks for the donations.

[audeojude]

yes.. single edit gives me back the interface in a second or so. bulk edit takes a long time.

Scott

[kovidgoyal]

At the end of every bulk edit, the database is reset, which takes as much extra time as starting up calibre.

EDIT: There was an additional delay introduced by reselecting the previously selected rows, this will be fixed in the next release.

[audeojude]

Thanks.. good to know what's going on.