Calibre Companion test for very large libraries

[Sefiriot]

Originally Posted by chaley

As DoctorOhh said, sending from calibre to CC has been optimized for some time. Calibre only sends metadata to CC that has changed (has a later last_modified time stamp).

I have been working for some time on doing the same thing in the other direction, CC -> calibre. There are some gotchas having to do with multiple libraries and CC's database performance. For example, most of the time required to send metadata up to calibre is used in CC reading book metadata from database 1 by 1. The network time is overlapped with the db time. Because of this, the only way to get significant improvement is to read everything required at one go from the db, but that can't work if more than a little bit of memory is required because these android devices are so memory challenged.

At any rate, this week I came up with an implementation that works: reads the DB in one query but requires minimum memory on the device (around 60 bytes per book). Even this might be too much for some of the devices out there, but we will see. The calibre changes will be in tomorrow's release, and I will start another prerelease cycle next week to test CC + calibre together.

In my experiments, with the new code the CC->calibre metadata sync time for 100 books fell from 6 seconds to 1/10 second. I haven't done any tests yet with more books, but I suspect that the time for 1000 books will be under 1/2 second. CC should run at close to network speed, sending around 100 bytes per book to calibre. CC finding calibre using automatic mode takes around 2 or 3 seconds and the initial cc/calibre handshaking adds around 2 more seconds, just as it does today. NB: this improvement will be seen after one full cycle with the new code available because calibre must build its cache.

Sounds very good, though to clarify, we are talking about internal device memory and not device RAM, yes? What would the minimum memory requirement be-- DB size + 20%?

[chaley]

Originally Posted by Sefiriot

Sounds very good, though to clarify, we are talking about internal device memory and not device RAM, yes? What would the minimum memory requirement be-- DB size + 20%?

Hard to know exactly what those words mean.

I am talking about RAM the application uses while it is running, i.e., the heap and the stack. I am not talking about the kind of memory where the database or books live.

From what I can tell, most devices give a running app at least 16MB of working memory. Some get more. Assuming 15,000 books (don't think you have got there but you are trying ), that means that the connection handshake process will consume around 900KB of heap instead of around 120KB. I have no idea if that difference will break something, but it easily could if a device is running at the edge.

It could especially be problematic with Gingerbread devices because garbage collection of bitmaps is basically broken in Gingerbread, causing CC to run out of memory at very inconvenient times. This is why we have so much trouble with devices like the Nook Color. Fortunately for us, at this point only about 5% of CC users run Gingerbread. ICS and up do a much better job.

[chaley]

I have some timings for using the caching system. These timings take into consideration the calibre overhead to do book matching and to determine whether any metadata must be sent back to CC.

The device is a slow Samsung Galaxy tab 10.1. Also, there are a zillion wifi hubs around me, so I don't get the best throughput that can be imagined.

100 books: cache off: 7.3 seconds. cache on: 2.4 seconds
500 books: cache off: 39 seconds. cache on: 4.6 seconds
1000 books: cache off: 60 seconds. cache on: 7.2 seconds

This seems to indicate that the time with the cache on is approximately 2 seconds plus .5 second per hundred books.

[DoctorOhh]

Sounds great!

Looking forward to testing this myself.

[chaley]

Originally Posted by DoctorOhh

Sounds great!

Looking forward to testing this myself.

Unfortunately while testing I found a show-stopper bug on the calibre side. The bug breaks displaying the device view. This is too fundamental to ignore even in a prerelease, so the test must wait until after next week's calibre 1.5 release.

[DoctorOhh]

Thanks for the update.

[Hobbit]

Originally Posted by DoctorOhh

CC, but you most likely can tell from my numbers that calibre does only send the metadata / covers back to CC that have changed. If you reconnect again I bet you see your "Send Metadata to Device" time plummet.

Oh I agree that calibre sends only the changes to calibre companion. I meant from CC to calibre.

I will try to clearly say what I meant regarding this during the weekend when I get a moment.

[Hobbit]

Originally Posted by chaley

This seems to indicate that the time with the cache on is approximately 2 seconds plus .5 second per hundred books.

Looks great.
Will look forward to the release!