View Single Post
Old 08-26-2013, 03:45 AM   #10
chaley
Grand Sorcerer
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 12,451
Karma: 8012886
Join Date: Jan 2010
Location: Notts, England
Device: Kobo Libra 2
Quote:
Originally Posted by Sefiriot View Post
Heh. Well my calibre library is already at 12k+. Let's hope CC won't fall over and die at 20k, which is where I'm pretty sure my library will be at in less than 6 months. Pretty sure the main memory isn't full, but could you please clarify what you mean by main memory? The Galaxy Note's internal memory structure is plain weird since there's the 2GB system storage, then the 11GB or so memory that seems to be what the system sees as USB storage (mnt\sdcard). Do you mean the 2GB internal? Currently all my books are on mnt\ext_sd\Books, which is the actual external SD. calibre's files are in mnt\sdcard, which has 6GB of free space left.
It is hard to know what I mean without handling the device and seeing what memory is used for what.

CC uses two memory locations: one for its database and settings and one for its books. The first is in the same place that the device stores apps, and the second is in visible storage (sdcard). It would not surprise me if CC's database is in the 2GB hunk. A database for 10,000 books could easily be 100 megabytes large, or more.
Quote:
I'm enclosing my log from this evening ...
OK, lots here.

First thing I see: the first connection failed before sending any metadata up to calibre. This is consistent with the "scan for books" problem, where the scan takes longer than 60 seconds. One reason this can happen is if CC is storing books in a folder where other things are being kept. In one case we saw, the person set the root of the SD card as CC's folder, so the scan was walking the entire SD card. If I read your post correctly, you changed the setting between the first and second connect, and the second connect worked properly.

Another thing I see is that the network transfers are very slow. I am seeing 15 to 60 seconds per 100 books when uploading metadata. On all of our devices that number is 2 to 4 seconds. I have no explanation for this difference.

Looking at the times to upload books, they also are way out of line. On our Tab 10.1, the slowest device we own, sending a 400 MB book file takes around a half second. On your device I am seeing 4 to 30 seconds.

Things went south at line 4180 of the log, where we see our first timeout. There is no indication why the timeout occurred, but no transfers after that point worked until you reconnected at line 4390. At this point the time to upload metadata has gotten substantially worse, with all the times around 60 seconds per 100 books. This connection attempt aborted 2800 books into the "send metadata" cycle.

The next connection attempt, at line 4496, is interesting because the times to send metadata has gone back to what it was in the first connection. Do you have any idea why? Did you reboot your device? Or something else? I am also confused about why calibre felt it needed to send metadata for 292 books. Had you changed tags or otherwise modified books?

We also see the same slowdown when sending metadata back. On my tab 10.1 it takes between 100 and 200 milliseconds per book. On you device it is taking 1 to 2 seconds per book. As before, I have no idea why this would be.

Things broke again at line 7547, with another timeout. Looking at the transactions above the timeout, it seems clear that something happened on the CC side. What, exactly, is a mystery.

So, where does all of this leave us? My guess is that your device is being very challenged, either in working memory or storage or both. The long network times could be caused by the device running out of working memory and needed to "garbage collect" (go find memory that isn't used any more). If storage is being challenged, then the database update would run extremely slowly because the device must do operations carefully and in an order that doesn't exacerbate the situation.

The above theory is consistent with your database being corrupted. Possibly at some point the database manager on your device was unable to do an update and was unable to roll back previous updates, a situation what would destroy CC's database. If you can determine how much of the 2GB of "system memory" is being used, then check if it is near the limit. Also, look at the "app data" for CC in settings to see how much memory it is using. These numbers do not count the books.
Quote:
ETA: Tried syncing CC after crash with calibre wiped out library on device (book files are all intact though). It partially synced (got around 90 books) then failed when sending metadata to calibre with this message:

Code:
'NoneType' object has no attribute 'lpath'

Traceback (most recent call last):
  File "site-packages\calibre\gui2\device.py", line 85, in run
  File "site-packages\calibre\gui2\device.py", line 476, in _books
  File "site-packages\calibre\devices\smart_device_app\driver.py", line 49, in _synchronizer
  File "site-packages\calibre\devices\smart_device_app\driver.py", line 1033, in books
  File "site-packages\calibre\devices\smart_device_app\driver.py", line 693, in _set_known_metadata
AttributeError: 'NoneType' object has no attribute 'lpath'
Will provide a complete debug log later once this latest sync attempt finishes-- I'm letting it run while I go to bed since it's closer to dawn that I'd like right now.
This error means that CC's database is not in good shape. It sent empty metadata to calibre for the book in question. The only way I can think of where that can happen is that the db does not actually contain the metadata, something that can happen only if there is a problem while writing the info to the DB.

Bottom line: I am concerned that you are pushing your device and CC beyond its limits. There is no way that CC can survive database corruption introduced by Android for whatever reason. It is also difficult for CC to survive slowdowns required by Android to "maintain" the database.

One thing we can do is up the timeout, but I don't think that will help anything.
chaley is offline   Reply With Quote