Connection performance improvement?

[chaley]

A change was made in calibre 2.3 to reduce the time "waiting for calibre" before metadata is transferred. A user with 18 thousand books on his device (!) said that the time went from 12 minutes to 2 minutes.

The amount of improvement is directly related to the number of books on the device. There was a check that was done ((books on device) squared) number of times. For 18,000 books that is some 320 million checks. Five thousand books is 25 million checks.

Is anyone else seeing an improvement? I am wondering how many books must be on the device to make the change apparent.

[kaufman]

Not really noticeable with 5000 books. I wouldn't have know there was a change if you hadn't posted this.

[PO40600]

Time of connection has been considerably reduced using Calibre 2.3, compared to 2.2.
I'm the user you talk about

But, unhappily, 2.4 release has broken all your changes. Time of connexion is twice worst than in 2.2 : 26 minutes for initial connection, and the same after transfert of 38 books.

You told me that it would be fixed in 2.5.

BTW thank you for your quick answer.

Edit : When writing this message, I hadn't read the pinned message. Thanks for all

[Hobbit]

I have just over 3000 books on my Galaxy GT-800 tablet. Calibre library has over 5600.
The initial steps have become fast compared to Calibre 2.4 (timings: Get Device Information (12s), set library information (1s) and Get List of Books on Device(46s) and Sending Metadata to Device (27m 16s for 622/2280) )

The metadata update is very slow. (roughly 15 items per 50 seconds)

Q. Can't this metadata update happen only for changed items? Or optionally can the user turn this feature off?

Best wishes.

[chaley]

Originally Posted by Hobbit

I have just over 3000 books on my Galaxy GT-800 tablet. Calibre library has over 5600.
The initial steps have become fast compared to Calibre 2.4 (timings: Get Device Information (12s), set library information (1s) and Get List of Books on Device(46s) and Sending Metadata to Device (27m 16s for 622/2280) )

The metadata update is very slow. (roughly 15 items per 50 seconds)

Q. Can't this metadata update happen only for changed items? Or optionally can the user turn this feature off?

Best wishes.

First: your questions: can updating metadata happen for only changed items? Yes. This is what is does today. Can updating metadata be made optional. No, it can't for the wireless device connection. The content server connection does not update metadata, so you can use that to download books without that problem.

Now, to the timings. There is something very wrong.

As noted above, metadata is sent to the device only if has changed. For my "production" tablet (Nexus 7 2012) with 1250 books on it, if the metadata hasn't changed then the times you mention for me are:
- Get device info: 2 seconds
- Set library info: 0 seconds
- Get list of books: 7 seconds
- Send metadata: 1 second.

Note that metadata is considered changed if the last_modified date for the metadata changes. There are several ways that the date can change including editing templates of custom columns or requesting that all metadata be backed up. It can change for books if you change the case of a tag or change a series.

Get device info should not take 12 seconds. The only thing calibre does is read its cache file and send a few packets to CC. The long time makes me wonder if you are running into memory problems on the calibre machine, causing reading the cache to be very slow.

As an experiment I changed the template for a custom column, which changed the last_mod time for every book. The time to send metadata was 12 minutes, or 0.58 seconds per book. I did it twice and got the same number each time. My number is consistent with what others have told me they see. It is 5 times faster than you are seeing. The question is why? Again, I wonder if calibre is memory constrained. If it is then the process of determining what metadata to send would be going to virtual memory, something that would slow down everything.

What are the specs of the computer you are using? If you are running windows, are you running the 64-bit calibre?

I also would wonder about the speed of your wifi, but slow wifi would not explain the long get device info times.

[kaufman]

I have noticed one thing. It does the "sending metadata to calibre" where it sends all of the metadata a lot more often.

[Hobbit]

Originally Posted by chaley

Now, to the timings. There is something very wrong.

What are the specs of the computer you are using? If you are running windows, are you running the 64-bit calibre?

I also would wonder about the speed of your wifi, but slow wifi would not explain the long get device info times.

Hardware: Acer Aspire 7739Z
Rating: 4.6 Windows Performance Index
CPU 5.7
Memory 7.1
Graphics 4.6
Disk transfer Rate 5.9
Installed RAM: 8 GB (7.68)
System Type: Windows 7 Professional with SP1. 64bit

Calibre 2.5 64bit

Physical Memory: 7860MB
Cached: 2806MB
Available: 4890MB
Free: 2100MB

Number processes: 85

Wifi: 50MBPS

Hope some of this may help you to make sense
thank you!

[Hobbit]

Originally Posted by chaley

Get device info should not take 12 seconds. The only thing calibre does is read its cache file and send a few packets to CC. The long time makes me wonder if you are running into memory problems on the calibre machine, causing reading the cache to be very slow.

As an experiment I changed the template for a custom column, which changed the last_mod time for every book. The time to send metadata was 12 minutes, or 0.58 seconds per book.

I still get 12 seconds to Get Device Info. The memory picture at that time was:
(all MB)
7860 total/2953 cached/4311 available/1372 free

Processes: 92
Commit (GB) 3/15

Kernel Memory (MB) 407 paged/142 Non paged

The send metadata is again very slow. More than 2 seconds per item. Which is more than double the amount of time it takes for your Nexus. The memory picture remains essentially the same and the number of processes is 91.

[chaley]

Originally Posted by kaufman

I have noticed one thing. It does the "sending metadata to calibre" where it sends all of the metadata a lot more often.

I don't see any reason why this would have changed.

There are several things that force CC to resend metadata to calibre

1. Clearing cc's data or reinstalling. That causes CC to get a new unique ID, invalidating the old cache.
2. Using more than one computer. CC's cache is kept on a per-machine basis. If you change books on one machine then metadata for those books will be sent to calibre when you connect to some other machine. This is true even though the date of the book on the device matches the date of the book in the library, because it does not match the date of the book in the per-machine cache.
3. Deleting the cache file. I have no idea where the cache files are stored on macs or if there is something that will "clean up" from time to time. On windows they are stored in \Users\{userName}\AppData\Local\calibre-cache, and to my knowledge are never auto-cleaned.
4. Killing calibre with a device connected. If calibre isn't given the opportunity to shut down the connection with CC then the cache will not be written.

For example, assume you use two machines (and I know you do), and that the libraries are synced somehow (don't know if you do this). Further assume that on machine 1 change the template of a custom column, create a new custom column, trigger a metadata backup, or change CC's cover size. Now connect CC. Calibre will send all metadata to CC. Now go to machine 2 and connect CC. CC will send all the metadata to calibre because what it has is not the same as in the cache. However, calibre should send no metadata back to CC because the newly cached copy and the copy in the library already match.

Does any of this match what you are seeing?

[chaley]

Originally Posted by Hobbit

I still get 12 seconds to Get Device Info. The memory picture at that time was:
(all MB)
7860 total/2953 cached/4311 available/1372 free

Processes: 92
Commit (GB) 3/15

Kernel Memory (MB) 407 paged/142 Non paged

The send metadata is again very slow. More than 2 seconds per item. Which is more than double the amount of time it takes for your Nexus. The memory picture remains essentially the same and the number of processes is 91.

At this point the only way I can proceed is to see debug logs for both calibre and for CC. See the FAQ answer How do I get help? for instructions on obtaining and sending a debug log. Send the log to "cc_debug_logs [at] charles [dot] haleys [dot] org" (making the obvious substitutions). Also please remove the multipie address when you send a CC debug log.

[kaufman]

Originally Posted by chaley

1. Using more than one computer. CC's cache is kept on a per-machine basis. If you change books on one machine then metadata for those books will be sent to calibre when you connect to some other machine. This is true even though the date of the book on the device matches the date of the book in the library, because it does not match the date of the book in the per-machine cache.

For example, assume you use two machines (and I know you do), and that the libraries are synced somehow (don't know if you do this). Further assume that on machine 1 change the template of a custom column, create a new custom column, trigger a metadata backup, or change CC's cover size. Now connect CC. Calibre will send all metadata to CC. Now go to machine 2 and connect CC. CC will send all the metadata to calibre because what it has is not the same as in the cache. However, calibre should send no metadata back to CC because the newly cached copy and the copy in the library already match.

Does any of this match what you are seeing?

Yes, I use two machines, and I am syncing my dropbox library via dropbox. Also, when I disconnect, I always use the disconnect button in CC. I never just kill the processes on either device. I am using the most recent version of both programs.

I am seeing things that are similar to what you say I should expect, but not exactly the same.

1) If I change a custom column so that all metadata is impacted, I will get a full metadata transfer from C to CC the next time I connect on both machines.

2) As a subset of this, if I change a few books, they will all resend their metadata on the next connect.

3) If I change the ePub, it will get sent on the next connect, but only once. Not once per machine.

4) Each time I connect on either machine for the first time after connecting on the other machine, I get the "asking calibre what metadata it needs" for about 20 seconds, then I get the "sending metadata to calibre" where it sends all of the metadata for my whole library. This takes over a minute. Then it resends whatever metadata has changed on the other machine.

5) If I then connect on either machine where I haven't connected on the other machine, I get the "asking calibre what metadata it needs" for about 20 seconds. Then it sends whatever metadata has changed on the this machine.

This might not be a new issue. Connections do seem a lot slower these days, but I can't tell for sure when it started. I haven't bought it up because it was just an inconvenience, and I didn't wan't to bother you if everything was still working. I also suspected it might have something to do with my large library

[chaley]

Originally Posted by kaufman

Yes, I use two machines, and I am syncing my dropbox library via dropbox. Also, when I disconnect, I always use the disconnect button in CC. I never just kill the processes on either device. I am using the most recent version of both programs.

I am seeing things that are similar to what you say I should expect, but not exactly the same.

1) If I change a custom column so that all metadata is impacted, I will get a full metadata transfer from C to CC the next time I connect on both machines.

This is as expected. It should be sending metadata from C to CC, not from CC to C.

2) As a subset of this, if I change a few books, they will all resend their metadata on the next connect.

As expected.

3) If I change the ePub, it will get sent on the next connect, but only once. Not once per machine.

Do you have book sync turned on? I hope so.

This too is as desired, but I hadn't actually tried it. What it means is that dropbox is preserving file modification dates for the formats as it syncs them. This is good.

4) Each time I connect on either machine for the first time after connecting on the other machine, I get the "asking calibre what metadata it needs" for about 20 seconds, then I get the "sending metadata to calibre" where it sends all of the metadata for my whole library. This takes over a minute. Then it resends whatever metadata has changed on the other machine.

This one I don't understand. It happens *every time* you change the computer running calibre? I tried this using my laptop and desktop and it does not happen for me, assuming I had not changed my library.

The only way I can see this happening is if calibre's cache of the metadata on the device is not being saved properly. If this happens then the first time you connect after starting calibre CC will send all metadata. It won't send it again until you quit and restart calibre. Is this what is happening?

I looked at calibre's code and on a mac the cache is in "~/Library/Caches" where the ~ character means your home folder. Does this folder exist? Is it synced between the two machines?

5) If I then connect on either machine where I haven't connected on the other machine, I get the "asking calibre what metadata it needs" for about 20 seconds. Then it sends whatever metadata has changed on the this machine.

This again is normal. During that 20 seconds you will see "Analyzing book info" (or some such) on the calibre side.

This might not be a new issue. Connections do seem a lot slower these days, but I can't tell for sure when it started. I haven't bought it up because it was just an inconvenience, and I didn't wan't to bother you if everything was still working. I also suspected it might have something to do with my large library

The only thing I have done that will have slowed down the "analyzing" phase is book syncing. If that option is on then calibre must check the date of every book file, which requires going to the disk. That could account for the 20 seconds being more than it was before.

[kaufman]

Originally Posted by chaley

This is as expected.

Just so you know, all the things that were as expected were expected by me as well. I included them for completenessness (is that a word) and so you would know what was working.

Originally Posted by chaley

The only way I can see this happening is if calibre's cache of the metadata on the device is not being saved properly. If this happens then the first time you connect after starting calibre CC will send all metadata. It won't send it again until you quit and restart calibre. Is this what is happening?

I looked at calibre's code and on a mac the cache is in "~/Library/Caches" where the ~ character means your home folder. Does this folder exist? Is it synced between the two machines?

Aha, that explains it. The only thing I sync across devices is the book folder. Is there some reason that the Cache isn't/can't be stored in there as well?

Originally Posted by chaley

The only thing I have done that will have slowed down the "analyzing" phase is book syncing. If that option is on then calibre must check the date of every book file, which requires going to the disk. That could account for the 20 seconds being more than it was before.

Aha again. I tried turning off the copy new books, and the time drops from 20 seconds to about 5 seconds. Rather then checking the date each time, couldn't Calibre cache the last changed date and just compare that to the last sync date?

Is anyone updating their book files outside of Calibre? If not, this should work.

[chaley]

Originally Posted by kaufman

Aha, that explains it. The only thing I sync across devices is the book folder. Is there some reason that the Cache isn't/can't be stored in there as well?

I was looking for something that might explain what you are seeing. I don't think that the cache should be synced. There are too many ways that things can get bonkers if the cache from one calibre instance is used in another.

Am I correct in assuming that the libraries on the two machines are in fact the same library, synced by dropbox? In particular, they share the same metadata.db file? And that metadata.db file has not been "conflicted"? If so then I can't find any reason that the following steps shouldn't work

1. On machine A, sync the device.
2. On machine A: change nothing else
3. On machine B: sync the device. This might or might not cause CC to send metadata to C.
4. On machine B: change nothing else. At this point the caches should contain identical content.
5. On machine A: sync the device. CC should not send metadata to C.

Aha again. I tried turning off the copy new books, and the time drops from 20 seconds to about 5 seconds. Rather then checking the date each time, couldn't Calibre cache the last changed date and just compare that to the last sync date?

Is anyone updating their book files outside of Calibre? If not, this should work.

Lots of people modify books outside of calibre, using various tools and editors. In addition, as far as calibre is concerned the various plugins that change the book file are external programs. And finally, calibre makes no attempt to keep a date for the book files themselves. So no, the only way to get the changed date for a book is to check the date of the file.

[kaufman]

Originally Posted by chaley

Am I correct in assuming that the libraries on the two machines are in fact the same library, synced by dropbox? In particular, they share the same metadata.db file? And that metadata.db file has not been "conflicted"?

You are indeed correct. In addition, I just ran the "Check Library" command and everything matches in both directions.

Originally Posted by chaley

If so then I can't find any reason that the following steps shouldn't work

1. On machine A, sync the device.
2. On machine A: change nothing else
3. On machine B: sync the device. This might or might not cause CC to send metadata to C.
4. On machine B: change nothing else. At this point the caches should contain identical content.
5. On machine A: sync the device. CC should not send metadata to C.

I'll try that and see. It might take me a few days, because the two computers are in different locations. Having said that, if I make a change on one of the computers, then the cache on the other computer will no longer match the contents of the database? What does Calibre do in that situation?

Originally Posted by chaley

So no, the only way to get the changed date for a book is to check the date of the file.

Got it. Well, its still worth having the automatic epub update, so I will live with the longer connect times.