MobileRead Forums - View Single Post - Calibre Companion test for very large libraries

chaley · 08-31-2013, 07:23 AM

@DoctorOhh and @Sefiriot:

Regarding the long send and the timeout:

I know why this is happening, and it is very surprising to me. The old implementation used a passel of tasks (threads), one per book, to update the database. This created a lot of contention, slowing things down and potentially creating problems. The new implementation uses a queue for requests to update the database for a book, ensuring that CC writes one book at a time to the database.

What we just learned is that the network is a *lot* faster than the database, as least in certain situations. What you are seeing is the queue getting so long that it cannot finish writing the books to the database in the 60 seconds that calibre is willing to wait. From my timing tests I thought that would require CC to get more than 200 books behind, which I thought was impossible. Well, apparently it isn't.

I will fix this by not permitting the queue to get longer than X, probably 4 or 5. If it gets that long, CC will wait for a book to get to the database before it adds another to the queue. This will slow down the apparent network transfer speed, but will ensure that when the transfers finish there isn't minutes of work left to do. @DoctorOhh, this will restore the network slowdown you previously saw, but I hope to a lesser degree.

Regarding the bug in the cleaning progress dialog: thanks. I have fixed that.

Regarding the speedup in grouping: this must be "caused" by the database clean because I made no changes in that code. The cleaning process creates a new database with everything packed together in a nice order and all empty space released. My guess (close to a certainty) is that the "nice order" is permitting the database queries to run substantially faster, giving you the performance improvement.

Finally, I am adding the ability for CC to write its own debug log. There will be a preference that when checked causes CC to log *lots of stuff* to the file calibre.companion.debug.log.txt in the root folder of the device public storage. It is there because where logging happens I cannot get any application-specific configuration, so I can't put the file into CC's default books folder. Oh well...

08-31-2013, 07:23 AM	#11
chaley Grand Sorcerer Posts: 12,461 Karma: 8025600 Join Date: Jan 2010 Location: Notts, England Device: Kobo Libra 2	@DoctorOhh and @Sefiriot: Regarding the long send and the timeout: I know why this is happening, and it is very surprising to me. The old implementation used a passel of tasks (threads), one per book, to update the database. This created a lot of contention, slowing things down and potentially creating problems. The new implementation uses a queue for requests to update the database for a book, ensuring that CC writes one book at a time to the database. What we just learned is that the network is a lot faster than the database, as least in certain situations. What you are seeing is the queue getting so long that it cannot finish writing the books to the database in the 60 seconds that calibre is willing to wait. From my timing tests I thought that would require CC to get more than 200 books behind, which I thought was impossible. Well, apparently it isn't. I will fix this by not permitting the queue to get longer than X, probably 4 or 5. If it gets that long, CC will wait for a book to get to the database before it adds another to the queue. This will slow down the apparent network transfer speed, but will ensure that when the transfers finish there isn't minutes of work left to do. @DoctorOhh, this will restore the network slowdown you previously saw, but I hope to a lesser degree. Regarding the bug in the cleaning progress dialog: thanks. I have fixed that. Regarding the speedup in grouping: this must be "caused" by the database clean because I made no changes in that code. The cleaning process creates a new database with everything packed together in a nice order and all empty space released. My guess (close to a certainty) is that the "nice order" is permitting the database queries to run substantially faster, giving you the performance improvement. Finally, I am adding the ability for CC to write its own debug log. There will be a preference that when checked causes CC to log lots of stuff to the file calibre.companion.debug.log.txt in the root folder of the device public storage. It is there because where logging happens I cannot get any application-specific configuration, so I can't put the file into CC's default books folder. Oh well...