Thread: Calibre Companion test for very large libraries
View Single Post
Old 09-20-2013, 02:58 AM   #31
Sefiriot
Bujavid tekikin
Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.Sefiriot writes the songs that make the whole world sing.
 
Sefiriot's Avatar
 
Posts: 139
Karma: 40366
Join Date: Aug 2013
Location: Malaysia
Device: LG G3 D855, Samsung Galaxy Note N7000
Quote:
Originally Posted by chaley View Post
As DoctorOhh said, sending from calibre to CC has been optimized for some time. Calibre only sends metadata to CC that has changed (has a later last_modified time stamp).

I have been working for some time on doing the same thing in the other direction, CC -> calibre. There are some gotchas having to do with multiple libraries and CC's database performance. For example, most of the time required to send metadata up to calibre is used in CC reading book metadata from database 1 by 1. The network time is overlapped with the db time. Because of this, the only way to get significant improvement is to read everything required at one go from the db, but that can't work if more than a little bit of memory is required because these android devices are so memory challenged.

At any rate, this week I came up with an implementation that works: reads the DB in one query but requires minimum memory on the device (around 60 bytes per book). Even this might be too much for some of the devices out there, but we will see. The calibre changes will be in tomorrow's release, and I will start another prerelease cycle next week to test CC + calibre together.

In my experiments, with the new code the CC->calibre metadata sync time for 100 books fell from 6 seconds to 1/10 second. I haven't done any tests yet with more books, but I suspect that the time for 1000 books will be under 1/2 second. CC should run at close to network speed, sending around 100 bytes per book to calibre. CC finding calibre using automatic mode takes around 2 or 3 seconds and the initial cc/calibre handshaking adds around 2 more seconds, just as it does today. NB: this improvement will be seen after one full cycle with the new code available because calibre must build its cache.
Sounds very good, though to clarify, we are talking about internal device memory and not device RAM, yes? What would the minimum memory requirement be-- DB size + 20%?
Sefiriot is offline   Reply With Quote