Observations on Using CC, Android 6.01 and Windows 10

[gsparks2]

This is not a rant just a set of observations.

I just bought a Samsung Galaxy S7 Edge. It is a huge upgrade from my S4. (understatement of the year)

I run the latest and greatest 64 bit Windows 10 on my 7+ year old PC.

I run the latest and greatest Calibre v 2.55 64 bit.
I run the latest and greatest CC v5.1.1.3

On my S7 I have a 64 gb SD card. This is where my huge book collection 20k+ resided on my S4.

I decided to delete and reinstall CC and my library on the SD card. This would give me a chance to utilize the CC local cloud feature. It would also allow me to get away from the Kit Kat enforced SD card folder for my books.

I removed the SD card from the phone and put it in a reader on the PC. I copied the Calibre Library to a folder in the root of the SD card. Fast, clean and efficient. I deleted the Kit Kat folder on the SD card that contained the CC book files. I reinstalled the SD card in the phone.

I connected CC to the Library on the card. I love the interface! Great job, Chaley!

I thought, this is wonderful, I can just use a sync application to keep the Calibre Library on the SD card up to date with the Calibre Library on my PC.

Then I discovered that features of Windows 10 and Android 6.01 prevent the PC from seeing the SD card on the phone as a drive. Bummer. I would have to remove the SD card from the phone and place it in the PC reader to use the syncing software. I am reluctant to do that because of wear and tear on the SD card tray on the phone and the issues that some people have had with the S7 Edge SD card staying seated.

So, my next great idea was to use CC to download the books from the SD card Cloud Calibre Library into the CC book folder then use the wireless connection to keep the CC library up to date with the PC Calibre Library. Then I could delete the SD card Cloud Calibre Library from the SD card.

I commenced doing that yesterday. I found the "On Device: No" folder in CC and used Download All.
Observation: It takes about 7 seconds to download each book from the SD Card Cloud Calibre Library to CC. I assume that CC is downloading the book file, the metadata file and adding the metadata to the CC metadata database and coping the book file to the CC library folder.

I then discovered some consequences of the features that Android has. These are killing of background processes and active apps. The download process will stop whenever Android decides that CC needs to stop. No messages, no app closing, it just stops. This requires frequent restarting the download process. It is not a start it an forget action.

There is a handy dandy notification that shows a red icon in the notification bar at the top of the phone. swiping down to see the list of notifications, there is a nice line that shows how many books remain to be downloaded. This notification is persistent even when CC has stopped the download, disconnected from the SD Card Cloud Calibre Library or even if CC is closed manually.
This number of remaining books to be downloaded never matches the number in CC when I refresh the On Device/No folder. It always off by 4 or 5 books.

I seemed to remember the last time I installed the CC library on the S4 using only the wireless connection that while slow, it was much faster than 7 seconds per book. During one of the stoppages, connected as a wireless device and used Calibre to push a hundred books to CC. This turned out to be no faster than the SD Card Cloud Calibre Library download. about the same or slower rate.

I project about 3 weeks to get all of the books into CC library at this rate.

I know I am pushing CC to do more that it is designed to do. I know that the phone is not designed to crunch massive databases for long periods of time.

All that being said all of the software and devices I am using are great and I appreciate the efforts made by so many people to develop them.

End of observations.

Suggestions or comments are welcome.

[chaley]

Originally Posted by gsparks2

So, my next great idea was to use CC to download the books from the SD card Cloud Calibre Library into the CC book folder then use the wireless connection to keep the CC library up to date with the PC Calibre Library. Then I could delete the SD card Cloud Calibre Library from the SD card.

I commenced doing that yesterday. I found the "On Device: No" folder in CC and used Download All.
Observation: It takes about 7 seconds to download each book from the SD Card Cloud Calibre Library to CC. I assume that CC is downloading the book file, the metadata file and adding the metadata to the CC metadata database and coping the book file to the CC library folder.

If you have custom columns "built from other columns" then you can speed this up a lot by installing this calibre plugin. It is available on the plugins manager. Do note that closing calibre or changing libraries will take longer if you install the plugin. How much longer I can't say, but I suspect that it is in the range of 1 second per 1000 books in the library.

I seemed to remember the last time I installed the CC library on the S4 using only the wireless connection that while slow, it was much faster than 7 seconds per book. During one of the stoppages, connected as a wireless device and used Calibre to push a hundred books to CC. This turned out to be no faster than the SD Card Cloud Calibre Library download. about the same or slower rate.

Now this is wrong. I would expect from .5 to 2 seconds/book. Could you send me a debug log so I can look at what might be going wrong?

Random thought: be sure that your phone's wifi connection is as good as it can be. When downloading books using the wireless device connection, network speeds matter.

[gsparks2]

I sent the debug file just now. Thank you!

Let me know if I need to recreate events so that it is captured in teh debug log.

[gsparks2]

I installed the plugin you suggested. I am not sure if I have custom columns "built from other columns" but I do have "pages", "Reading List" and "Formats" in my Calibre Library. Retried wireless connection and download of 30 books has taken 14 minutes.

I sort of tested the WiFi speed by comparing internet connection speeds of my PC which has an ethernet connection to the internet and the internet speed of my phone that has a WiFi connection to the internet using the same router device as the PC Ethernet. The download and upload speeds of the two devices are very close to the same, ~12.5 Mbps download and ~2 Mbps upload. I expect the WiFi speed to be much higher than the internet speeds based on what I have read. This test told me nothing of the value my WiFi speed, just that the WiFi speed is not slower than the internet connection speed.

[chaley]

Originally Posted by gsparks2

I sent the debug file just now. Thank you!

Let me know if I need to recreate events so that it is captured in teh debug log.

Yea, the wireless device connection stuff has scrolled off the bottom. If you could do a connect, transfer a few books, and then send me a debug log, I would appreciate it.

That log *does* contain lots of downloads from the cloud. It is taking 30 to 50 seconds per book!

[chaley]

Originally Posted by gsparks2

I installed the plugin you suggested. I am not sure if I have custom columns "built from other columns" but I do have "pages", "Reading List" and "Formats" in my Calibre Library. Retried wireless connection and download of 30 books has taken 14 minutes.

That is just wrong. The wireless device connection should never take thirty seconds per book. The cloud connection might.

I sort of tested the WiFi speed by comparing internet connection speeds of my PC which has an ethernet connection to the internet and the internet speed of my phone that has a WiFi connection to the internet using the same router device as the PC Ethernet. The download and upload speeds of the two devices are very close to the same, ~12.5 Mbps download and ~2 Mbps upload. I expect the WiFi speed to be much higher than the internet speeds based on what I have read. This test told me nothing of the value my WiFi speed, just that the WiFi speed is not slower than the internet connection speed.

The wireless device connection runs over the LAN. It should be a lot faster than over the internet.

[gsparks2]

I sent a fresher debug log.

[chaley]

Originally Posted by gsparks2

I sent a fresher debug log.

Well, this is a mystery. What the debug log says is that the network is incredibly slow. It is indeed taking nearly 30 seconds to send a book. By "network" I mean the time between where calibre starts to sends a book and CC finishes receiving it. I would expect this time to be less than a second, but it is instead running close to 30 seconds. It appears that the delay is in the actual network time used to send the book, but I am not sure of that.

Experience says that probably one of three things is happening:
1) Calibre is slow, perhaps because other things are happening on the computer that cause calibre to be swapped out. Be sure that the machine running calibre is not also running other stuff in the background.
2) Your network is very slow, perhaps caused by where your wireless router is in relation to the phone. This can be caused by interference from neighbor's wifi or by interference from DECT-like portable phones. Get your phone as close as you can to the wifi router and see if things get better.
3) Your device's network is very slow. This could be caused by other "stuff" running in the background on the device. Try rebooting the device to see if things work better.

[gsparks2]

I tested the SD card read/write speed using SD Tools. It showed 1560 MB/s writing speed and 1057,0 MB/s reading speed.

[gsparks2]

Originally Posted by chaley

Well, this is a mystery. What the debug log says is that the network is incredibly slow. It is indeed taking nearly 30 seconds to send a book. By "network" I mean the time between where calibre starts to sends a book and CC finishes receiving it. I would expect this time to be less than a second, but it is instead running close to 30 seconds. It appears that the delay is in the actual network time used to send the book, but I am not sure of that.

Experience says that probably one of three things is happening:
1) Calibre is slow, perhaps because other things are happening on the computer that cause calibre to be swapped out. Be sure that the machine running calibre is not also running other stuff in the background.
2) Your network is very slow, perhaps caused by where your wireless router is in relation to the phone. This can be caused by interference from neighbor's wifi or by interference from DECT-like portable phones. Get your phone as close as you can to the wifi router and see if things get better.
3) Your device's network is very slow. This could be caused by other "stuff" running in the background on the device. Try rebooting the device to see if things work better.

Thank you. I will do that and let you know.

[gsparks2]

After much investigation I have determined that my vintage network adapter on my PC is very slow. No newer drivers for Windows 10 are available.
Just for your info:
I stopped all other programs except for Calibre.
I shutdown and restarted my Internet access device and the PC.
These did not change the wireless connection download speed into CC. Using Resource Monitor I could see very slow LAN transfer speeds.

Thank you for your help.

[chaley]

Random thought: if you are willing to take out the SD card and put into the card reader *once* then you can do the initial transfer of all your books more more quickly. Mount the SD card in the card reader then copy your entire calibre library to the card. Be sure that the metadata.db file is copied. It wouldn't be silly to try to open the copied library with calibre just to be sure.

Next, put the SD card back into your device and open it with CC's "Local library" cloud connection. You can then transfer books at SD card speed, which (I think) will be around 1/3 to 1/2 second per book, the time it takes CC to update its database.

Another random thought: try using the content server connection. It uses the network quite differently than the wireless device connection, which might result in better performance.

[Dopedangel]

instead of using a sync windows app over mtp once you have already copied the calibre folder to your device. You could probably use an android app like folder sync to sync over your home network. As only modifications are transfered shouldn't take long.

Sent from my SM-G925I

[gsparks2]

Originally Posted by Dopedangel

instead of using a sync windows app over mtp once you have already copied the calibre folder to your device. You could probably use an android app like folder sync to sync over your home network. As only modifications are transfered shouldn't take long.

Sent from my SM-G925I

Great idea! Thanks I will try it.

[crane3]

My observation is that transferring books from Calibre to the Android "internal memory" is quite faster than to the external SD card; noticeable when I sent down all the books in the library. Even with 3 or 4 books at a time, time to the SD is slower than to "memory"; with 1 book I did see a slight hesitation prior to the start of the progress graph.

A "security/permission" check each time for a book? There was a big difference in time when I first loaded CC & then changed CC lib to the SD card; but then it was also a difference in CC version also.