Calibre Companion test for very large libraries

[Sefiriot]

Originally Posted by DoctorOhh

If you tilted your phone a little and it slipped to landscape mode then the dialogue disappears even though the clean database function did not stop and is still functioning. The fix will be in the final release as well as the fix for calibre timing out after a large transfer.

Uhm, yeah, I know about that bug. Thing is, I had my phone upright in my hands, and the phone was flat on its back after and not once did it slip into landscape mode. Yet the dialogue vanished. It's too bad the CC cleaning function doesn't seem to appear in a list of running tasks in Android, AFAIK; be easier to tell if it is or isn't functioning.

In any case, I thought it was my mobile's antivirus at first, but nope. I put it down to possibly my phone or Android being momentarily quirky.

[chaley]

Originally Posted by Dhevi

I've configured to SmartDevice App Interface plugin and it is working as desired when connecting as a Wireless device and Sending to Device from Calibre, but I've noted that the same is not true when connecting to Content Server and downloading from within CC. I've not been able to find information as to if this is configurable to store files with the template: {author_sort}/{series}/{title}.

You cannot configure the file path used for books when receiving from the content server.

The reason: in general it is up to a web client (e.g., CC) to determine where to store downloaded files. Calibre does not provide any template for this because in the vast majority of cases the file path would be ignored. As such, CC has no information about where it is to store the file, so it uses author/title.

To fix this, either CC would need to implement the calibre template language (not going to happen -- way too hard), or calibre's content server would need to be extended to evaluate a template and send a suggested file path along with the metadata for a book. This second could happen, but given that it would be used only by CC and would probably need to be "per device", the justification for creating yet another point-of-maintenance in calibre is quite weak.

[chaley]

Originally Posted by Sefiriot

Here's another log testing CC 3.0.6. This time the transfer was about 1k+ titles. Interestingly metadata did finish sending this time before calibre timed out.

Since this is yet another monster log at 1.05 MB and over forum limits, it's again in my Dropbox.

Thanks. I will look it over to see if I can identify why the timeout happened.

I think that the "scan on connect" feature is beginning to having a more noticeable adverse impact during the initial handshaking period when CC and calibre are syncing up, now that the ondevice library's gotten this big (11560 titles).

Turn it off.

Scan on connect is useful for cases where the user sideloads books into CC's folder using some other app, for initial or complete reinstallations, or if the DB gets corrupted and the user clears it. In the normal course of events it should add nothing.

We now default it "on" because of the number of emailed questions the original dialog box provoked.

Ondevice database was at 199MB after this transfer before cleaning. Size after cleaning remains at 199MB. Took about 30+ minutes.

This matches my experience -- that frequently the db size does not change. However, subsequent db queries can get much faster because of the db "defragging" that cleaning does.

I wish I had some way to measure the defragging, and (even better) some way to know if it is necessary. Sigh.

I'd also like to report something strange: CC had been open since the file transfer (at least, not stopped) for several hours. Th phone of course had gone to sleep in between.

I then started CC's cleaning function. It brought up the "wait" dialogue screen which almost instantly disappeared and didn't clean the database. I then did a force stop and restart through device settings, told CC to clean the database, and all went back to functioning as usual. Possibly my device is just having a hiccup in functioning, but I thought you should know.

My guess is that the database was closed by Android while the phone was idle. The code in 3.0.6 didn't tolerate that very well (or should I say, not at all). I have changed CC to verify that the db is actually open on every query/action. I had been worried that this verification would slow CC down, but Android takes care of caching open database handles so in the normal case it costs almost nothing. This change will be in the next release/beta.

[chaley]

Originally Posted by Sefiriot

Here's another log testing CC 3.0.6. This time the transfer was about 1k+ titles. Interestingly metadata did finish sending this time before calibre timed out.

I looked at the log.

Interesting to note that the scan for books on the second connect (the one that worked) apparently took 25 seconds. That is a long time.

I have up'ed the timeout in calibre from 1 minute to 5 minutes. I considered getting rid of it all together, but decided that timing out does provide some information. The increase will be in the next calibre release.

Getting on to the stuff at the bottom, I am quite sure that this is caused by the same "queue length" problem we have seen before. Calibre thinks that CC has finished storing books, but in fact CC hasn't. It could be hundreds of books behind. This means that when the send metadata job starts (log line 6181), CC isn't listening, instead working its heart out trying to store metadata for books. A minute later, calibre gives up.

As I think I have said before, the next version will avoid this problem by refusing to queue saving metadata for more than 5 books. This lets CC overlap receiving metadata from calibre with writing the db, but doesn't allow it to get so far behind that timeouts are generated. It will appear to the user that the transfer is slower (faster than 3.0.5), but the overall time to send will not go up.

The next version will also use queuing when receiving metadata during the connection handshake. This should speed up metadata reception by 5% to 10% because network time will be overlapped with database time.

[Sefiriot]

Installed the new beta app you sent out chaley; performance has improved; many thanks. Sent you a CC debug log from the first run; if you need the calibre side logs as well say the word.

Meanwhile, I think that defragging the database makes grouping work faster; particularly since I have a boatload of tags. Display grouped by tags tends to slow down after a large send and speed up after I clean the DB, which I think ties in with what you were saying about faster DB queries after cleanup.

[chaley]

Originally Posted by Sefiriot

Installed the new beta app you sent out chaley; performance has improved; many thanks. Sent you a CC debug log from the first run; if you need the calibre side logs as well say the word.

Did you see any calibre timeouts?

The debug log went to my partner/son in England. He will forward it to me this morning. I look forward to seeing it and looking at the timings, and other things.

[chaley]

@Sefiriot: thank you for that debug log. Because of it I discovered that some datetime values coming from calibre are now arriving with the millisecond field set. This broke CC's date parser. Now that I know about it, I can fix it.

[Sefiriot]

A bit fuzzy on details, but no, no timeouts since the new version, I think. The last timeout I recall happening was before the new version was installed. Transfers are going amazingly fast compared to before: no more waiting half the morning to update my reading material!

[chaley]

CC V3.1, officially released today, includes all of the "big library" changes discussed in this thread.

[Merischino]

Hi,
Any chance I could join your test? I have 9751 books at latest count....

edit: please ignore, I just caught up with your latest post, Chaley -- looks like I will have it already.

[Sefiriot]

Oh, no worries Merischino, the more the merrier; and testing is still ongoing. I suspect chaley must think I'm deliberately out to find new and interesting ways to break CC/give him headaches by now, with as many reports and logs as I've submitted over the last few days.

Out of interest, are your 9k books also sitting on the Samsung Galaxy S3?

[Merischino]

Originally Posted by Sefiriot

Oh, no worries Merischino, the more the merrier; and testing is still ongoing. I suspect chaley must think I'm deliberately out to find new and interesting ways to break CC/give him headaches by now, with as many reports and logs as I've submitted over the last few days.

Out of interest, are your 9k books also sitting on the Samsung Galaxy S3?

No, maybe about 3000 are on the galaxy s3, and since my nexus 7 fhd doesn't have but the 16g I'm trying to keep my elibrary on it to under 100 books (which I find very challenging, even though i'm only reading the one book at a time...)

[Hobbit]

CC 3.1 on Galaxy Tab N-8000.

On wireless connect times taken:
Calibre: Get List of Books on Device: 5m 22s
Send Metadata to Device: 4m 51s

CC works flawlessly on the Galaxy Tab.

I wonder if there is any work around on the synchronization time between CC and Calibre. Probably Calibre and CC could each exchange a check sum and keep local files with device book lists (one each for each device...) and only exchange information that has changed. If check sums do not agree then a full fledged exchange could take as much time as it's needed. Just an idea.

Once again congratulations on a very fine and highly usable app!

[DoctorOhh]

Originally Posted by Hobbit

CC 3.1 on Galaxy Tab N-8000.

On wireless connect times taken:
Calibre: Get List of Books on Device: 5m 22s
Send Metadata to Device: 4m 51s

I have 10,000 books on my device and the numbers for me are as follows.

Get List of Books on Device: 4m 43s
Send Metadata to Device: 0m 2s

Originally Posted by Hobbit

I wonder if there is any work around on the synchronization time between CC and Calibre.

A lot of speed optimization has already been done, but I'm sure there is always room for improvement.

Originally Posted by Hobbit

Probably Calibre and CC could each exchange a check sum and keep local files with device book lists (one each for each device...) and only exchange information that has changed. If check sums do not agree then a full fledged exchange could take as much time as it's needed. Just an idea.

I don't know the inner workings of CC, but you most likely can tell from my numbers that calibre does only send the metadata / covers back to CC that have changed. If you reconnect again I bet you see your "Send Metadata to Device" time plummet.

[chaley]

Originally Posted by Hobbit

I wonder if there is any work around on the synchronization time between CC and Calibre.

Originally Posted by DoctorOhh

A lot of speed optimization has already been done, but I'm sure there is always room for improvement.

As DoctorOhh said, sending from calibre to CC has been optimized for some time. Calibre only sends metadata to CC that has changed (has a later last_modified time stamp).

I have been working for some time on doing the same thing in the other direction, CC -> calibre. There are some gotchas having to do with multiple libraries and CC's database performance. For example, most of the time required to send metadata up to calibre is used in CC reading book metadata from database 1 by 1. The network time is overlapped with the db time. Because of this, the only way to get significant improvement is to read everything required at one go from the db, but that can't work if more than a little bit of memory is required because these android devices are so memory challenged.

At any rate, this week I came up with an implementation that works: reads the DB in one query but requires minimum memory on the device (around 60 bytes per book). Even this might be too much for some of the devices out there, but we will see. The calibre changes will be in tomorrow's release, and I will start another prerelease cycle next week to test CC + calibre together.

In my experiments, with the new code the CC->calibre metadata sync time for 100 books fell from 6 seconds to 1/10 second. I haven't done any tests yet with more books, but I suspect that the time for 1000 books will be under 1/2 second. CC should run at close to network speed, sending around 100 bytes per book to calibre. CC finding calibre using automatic mode takes around 2 or 3 seconds and the initial cc/calibre handshaking adds around 2 more seconds, just as it does today. NB: this improvement will be seen after one full cycle with the new code available because calibre must build its cache.