[Android App] Calibre Companion Beta Bug

[manxam]

Alright, I think I may have figured it out. It's definitely a timing issue. I have both a 5ghz and 2.4ghz wireless in the house and the tab supports both. The 2.4 get's broader coverage and I use it for most of my non-crucial devices and the 5ghz I use for my laptop and heaver media devices.

The ping times to the tab using the 2.4ghz is .150ms or so.. With this I will run into connection issues during transfers but everything else works without issue (streaming movies, etc). The ping times to the tab using 5ghz are in the .075 to .09ms range and with this calibre will work without ever timing out.

A steady stream of pings to the tab during a calibre transfer while connected in either manner shows no missed pings but while connected at 2.4ghz and the slightly higher ping times calibre will eventually disconnect and give me the above errors. Transferring several hundred books over 5ghz I only had one such error and that's when ping times hit 0.155ms.

Perhaps I'm looking at this all wrong but it appears that calibre is being a little too aggressive and should have some wait states if it doesn't make it's connection right away or even some retry attempts?

[chaley]

Originally Posted by manxam

Here's the small debug log. Sent 8 random sci-fi books. Failed at sending the 6th. While the 6th book according to Calibre companion and Calibre actually did sync, there are only 5 books in the books folder (using file explorer) but there are 6 covers in Calibre Companion.

Ahhh! That explains a lot. The app didn't actually disconnect.

From what I can tell from the log and from googling around, MacOS has a networking bug where during sending a large amount of data it will occasionally report a "lack of resources" error instead of simply waiting to send the information. This is never supposed to happen, so I didn't account for the possibility. As far as I can tell it never does happen on windows or linux, and I don't have a Mac. Sigh...

I can work around the problem by changing how calibre writes to the network. Unfortunately that fix won't be available until Friday's calibre release.

Good ... your next post (#16) helps confirm the hypothesis. The faster wifi will return resources to the pool faster.

[chaley]

@manxam: I finished the calibre changes to work around the MacOS bug. Unfortunately I cannot test the changes because I cannot reproduce the problem

Fortunately, Kovid was kind enough to prepare an interim release of calibre containing my changes. Could you download and install http://calibre-ebook.com/downloads/b...bre-0.8.64.dmg (this is only for Macs!), then let me know if it helps? It would be so nice to know if the changes solve the problem before they are released for general consumption.

And again, thank you for your persistence, your patience, and your help!

[manxam]

Chaley, these changes are taking too long. I want my $3 back!

:P

Seriously though. I'm downloading the version now and I'll report my findings shortly.

Cheers!

[manxam]

Alright, I switched back to the 2.4ghz connection as it's used most often on this tab (best range in the house). Because I'm a glutton for punishment and I'm crazy I randomly selected 225 books and chose send to device.

0%.. all good
50% screen turned off..books continued to sync..
100%.. all good

Fixed. Excellent work Chaley.

Everything appears to be working like a champ here now. I'll continue to punish this thing and let you know what I find. I have several thousand more books that I can test with cuz.. umm.. you never know when you may need to read thousands of books back to back with no access to the internet??

Now for the feature question though I know it's already on your todo list: Show Metadata...

P.S. Tell Kovid he's awesome!

[chaley]

Originally Posted by manxam

Chaley, these changes are taking too long. I want my $3 back!

If I could give you the money back without also removing your google play license for the app, I would. It is so wonderful to have a patient user to work with. I *never* would have found this problem without your help.

I have been lucky on this forum. Most people seem to be interested in solving problems rather than scoring points. It is incredibly frustrating trying to solve mysterious problems at a distance. Having someone to work with makes all the difference.

[rita book]

My calibre program on my Macbook pro sometimes opens a foreign ip address, after starting it multiple time I will get a good local address. My Nexus 7 will connect and receive a book, the I receive a message "unfortunately, calibre companion has stopped" the only way to reopen the app is to uninstall and then reinstall, it will connect then after one download will stopped message.

[Adoby]

Another bug. Out of memory, if I interpreted the logs correctly.

I tried to upload 1200 ePubs to my Xoom (Jelly Bean). After around 580 ePubs CC crashed and keeps crashing even after reboots. Have managed to transfer a few more, around 600 ePubs, after that it seems that CC hardly works at all. Just crashes. Seems to be out of memory. I have sent logs using google feedback. Hope that is enough?

Plenty of room on the device. Seems to be RAM problem?

Exception class name: java.lang.OutOfMemoryError

[Mr. Goodbar]

Issue fixed with today's release.....thanks for the responsiveness.

Originally Posted by Adoby

Another bug. Out of memory, if I interpreted the logs correctly.

I tried to upload 1200 ePubs to my Xoom (Jelly Bean). After around 580 ePubs CC crashed and keeps crashing even after reboots. Have managed to transfer a few more, around 600 ePubs, after that it seems that CC hardly works at all. Just crashes. Seems to be out of memory. I have sent logs using google feedback. Hope that is enough?

Plenty of room on the device. Seems to be RAM problem?

Exception class name: java.lang.OutOfMemoryError

I've posted on the other thread and I think I'm having a similar issue. I was able to transfer around 176 books and then it crashed and will not restart. Every time I attempt to launch it force closes.

[chaley]

Originally Posted by rita book

My calibre program on my Macbook pro sometimes opens a foreign ip address, after starting it multiple time I will get a good local address. My Nexus 7 will connect and receive a book, the I receive a message "unfortunately, calibre companion has stopped" the only way to reopen the app is to uninstall and then reinstall, it will connect then after one download will stopped message.

First, I am away from a reasonable network and will not be able to reliably respond for a few days.

It sounds like you are having the same mac networking problem that several others have had and is described in post 18. Either try the interim release or waif for calibre's Friday release.

It is possible that your problem is caused by a bug triggered when a book did not have a calibre cover. If this is the case, the lastesf CC release 1.3 should fix it.

[chaley]

Originally Posted by Adoby

Another bug. Out of memory, if I interpreted the logs correctly.

I tried to upload 1200 ePubs to my Xoom (Jelly Bean). After around 580 ePubs CC crashed and keeps crashing even after reboots. Have managed to transfer a few more, around 600 ePubs, after that it seems that CC hardly works at all. Just crashes. Seems to be out of memory. I have sent logs using google feedback. Hope that is enough?

Plenty of room on the device. Seems to be RAM problem?

Exception class name: java.lang.OutOfMemoryError

You have most definitely pushed things far further than I expected. I am not surprised that it is running out of ram. CC loads the metadata into ram to do sorts. For performance reasons it loads cover thumbnails at the same time. Clearly if someone is manipulating 1000s of books we cant do it this way.

Next week when I have reasonable network access I will look at implementing cover caching. That will help keep performance acceptable for mord normal use cases while extending the number of books that can be kept in the device's library beford running out of memory.

[chaley]

Originally Posted by Mr. Goodbar

Issue fixed with today's release.....thanks for the responsiveness.

You are welcome.

[Adoby]

Originally Posted by chaley

You have most definitely pushed things far further than I expected. I am not surprised that it is running out of ram. CC loads the metadata into ram to do sorts. For performance reasons it loads cover thumbnails at the same time. Clearly if someone is manipulating 1000s of books we cant do it this way.

Next week when I have reasonable network access I will look at implementing cover caching. That will help keep performance acceptable for mord normal use cases while extending the number of books that can be kept in the device's library beford running out of memory.

OK. Perhaps use the database functionality in Android to store metadata? That way you could allow many more books, and also speed up start of the app? The transfer is rather slow so it should allow plenty of time to add/uppdate data in the database as each book is transfered.

I keep almost my entire ebook collection on my Xoom. At least the "good" part of it. 1200 was the fiction ePubs. Then there is the non-fiction and the PDF:s.

Today I use an external 32 GB Micro SD-card that I remove and update from Calibre. Would be nice to be able to update via WiFi instead...

[Steve_Haley]

Originally Posted by Adoby

OK. Perhaps use the database functionality in Android to store metadata? That way you could allow many more books, and also speed up start of the app? The transfer is rather slow so it should allow plenty of time to add/uppdate data in the database as each book is transfered.

Databases are actually rather slow, because it has to read from the device's hard-drive equivalent (flash storage). This is made worse by some phones/tablets having a restriction where only one app across the entire device can be accessing the flash drive. This means apps can end up waiting for the other ones to finish before being able to do their read/writes. Anyway, we do actually use the database, but we also store all the information in RAM for much quicker access. What Charles was saying was that we store *all* the info in RAM, including the thumbnails, and clearly that can't work with 1000+ books. We're going to have to change our system and make it be more choosy for this to work.

Regarding the transfer speed - we're also working on improving that. Maintaining compatibility with all desktop systems and all devices is a bit tricky, so please be patient

[Adoby]

Perhaps CC could be split into two parts? One really stupid part that only handles the actual connection and file/data transfers and syncs, and one more clever part that provides the book/metadata browser and interface to Callibre Content server?

Still, CC shows great promise, even if I can't use it currently.