[Alpha] Kobo-UNCaGED - Calibre wireless connection

[sherman]

Oof, that's some high memory usage!

To be honest, the metadata.calibre file isn't really the best way to be handling metadata on device. It's bloated, and difficult to efficiently work with on device. The main reason for using it is to keep in sync with the kobo driver in Calibre.

Thanks for the feedback so far. I'm trying to think of ways to better manage memory usage.

EDIT: Don't worry about

Jan 3 10:18:33 nickel: ( 1079.309 @ 0x3541df0 / packetdump.warning) "http://127.0.0.1:8181/messages" => "Connection closed"

That's normal, just what happens when the KU application closes, and the web browser is trying to access a connection that no longer exists.

[NiLuJe]

@sherman: Yeah, funnily enough, @pazos has spent the day dealing with similar fallouts from giant metadata caches in KOReader ;p.

It's... not fun to work with trivially once you start hitting those gigantic libraries on 512MB of RAM :/.

(Hell, I remember people managing to crash 32-bit builds of Calibre because they hit 2GB of allocs ;p)

[bigwoof]

Originally Posted by NiLuJe

ssh/telnet slowing down to a crawl while you're saturating the Wi-Fi with... Wi-Fi transfers makes perfect sense, FWIW.

Hi, I thought of this as well but no books are being transferred. it's possible metadata is so maybe. but the feel (on my side) seems like intense memory pressure causing the slugginess. that would explain why htop refuses to even start.

Originally Posted by NiLuJe

It'd also be consistent with a system going OOM, which makes this a bit tricky to diagnose...

yeah, I've tried replicating this so many times now and can't get it to crash while I'm monitoring the syslog. it's quite a fun puzzle.

ku crashes in so many different ways (inluding one where the Kobo printed out in big lettering KOBO DISCONNECTED on the ku "before connecting to calibre mostly blank page) and then the ku "disconnected" message appeared.

Originally Posted by NiLuJe

Enter USBNet! That way you'd get rid of the "is it Wi-Fi?" factor .

I'll try this but it really does not feel like the WiFi is being saturated at all..

Originally Posted by NiLuJe

EDIT: Re-reading the message, a reboot would imply either nickel hangs because the system is OOM, tries to swap, but can't because there's no swap, and that makes nickel so slow that it trips sickel's watchdog and forces a reboot. (Or nickel gets OOM-killed, sickel detects that, and reboots).

So what does the kernel say, exactly? Running klogd first means you get a timestamped kernel log in your syslog: an OOM-kill would be visible there.

There is nothing interesting in the syslog except for WiFi messages.. every now and then the kernel heat watchdog goes off. nothing interesting as far as I can tell. nothing that says that memory pressure is reaching dangerous levels. will keep trying

Originally Posted by NiLuJe

EDIT²: It's 4AM, so I'm a bit slow ;p. Is nickel still alive and responsive and how's the RAM usage when you happen to check the syslog and stuff just breaks at the first book?

4 a.m???? oof!! yeah, nickel is alive after ku dies. once it rebooted in nickel by itself and I had syslog running. but there was *nothing* in the syslog to say why it rebooted. just normal messages and ku was not running.. so a very wierd system instability reboot..

in htop when I start it with -s PERCENT_KERNEL, it appears that ku has a VIRT of ~830+M and a RES of ~200M but in all these cases, ku just disconnects before the first file was sent in calibre. these are not the case (that I first found) where files are being transferred, the memory usage explodes, htop and other operations crawl or just don't start, and kobo reboots.

also, the same memory and cpu usage was also shown for the nickel process as well btw. which makes sense as ku is spawned by nickel right?

and yeah, the libra has just 512M of which htop says ~480M is available

[bigwoof]

Originally Posted by sherman

Oof, that's some high memory usage!

To be honest, the metadata.calibre file isn't really the best way to be handling metadata on device. It's bloated, and difficult to efficiently work with on device. The main reason for using it is to keep in sync with the kobo driver in Calibre.

Thanks for the feedback so far. I'm trying to think of ways to better manage memory usage.

It's a fun puzzle. will keep trying to get better crash reports. the inconsistency suggests OOM conditions though.

also noted about the file. I posted that also to point out that syslog is working and I am seeing the correct things :-)

In any case, It's Sunday morning where I am and it's time to play games with the little one and turn into "entertainment dad"... :-) :-)

[sherman]

Just a heads up, I've created a Github issue for this.

[sherman]

Ok, found a major culprit for memory usage. The way I have been doing cover thumbnail creation is... suboptimal.

Basically I was being "clever" by spawning thumbnail generation into it's own goroutine. Couple of problems. Book download is generally faster than cover generation, so the number of active "create thumbnail" goroutines climbs, each consuming memory. And the second problem is I never limited the number of goroutines that could be spawned.

I've changed the code to wait for the goroutine to finish before continuing on to the next book. Makes the transferring book process take longer, but causes less problems.

I'm going to continue seeing if I can reduce the thumbnail creation memory usage further.

[bigwoof]

Originally Posted by sherman

Ok, found a major culprit for memory usage. The way I have been doing cover thumbnail creation is... suboptimal.

Basically I was being "clever" by spawning thumbnail generation into it's own goroutine. Couple of problems. Book download is generally faster than cover generation, so the number of active "create thumbnail" goroutines climbs, each consuming memory. And the second problem is I never limited the number of goroutines that could be spawned.

I've changed the code to wait for the goroutine to finish before continuing on to the next book. Makes the transferring book process take longer, but causes less problems.

I'm going to continue seeing if I can reduce the thumbnail creation memory usage further.

would it help if you just kept track of which books were downloaded and then generate the covers for those books without an existing cover *after* it is all done? that way, you have constant memory usage processing 1 or more books with a fixed number of threads for the cover generation portion right?

[sherman]

I'm talking about the thumbnails used for the homescreen/library views. Calibre sends the cover as part of the metadata, so the goal is to resize them to the appropriate sizes for the Kobo software and save them, so I can release the memory ASAP.

[bigwoof]

Originally Posted by sherman

I'm talking about the thumbnails used for the homescreen/library views. Calibre sends the cover as part of the metadata, so the goal is to resize them to the appropriate sizes for the Kobo software and save them, so I can release the memory ASAP.

interesting! I always thought the Kobo was pulling the thumbnails out from the epub itself seeing how it slowly updates when you scroll through the list. and the epub thumbnail was resized appropriately before sending by the calibre kobo driver.

are the thumbnails actually stored on disk as well? can you just drop them and have the device pull it from the epub?

[sherman]

What happens is that the kobo software extracts the cover from ebooks, and caches them to disk with a specific directory structure and filename. Basically, what KU (and the calibre kobo driver) do is pre-seed that cache with thumbnails so the kobo software doesn't have to do it.

KU actually tells calibre what size cover to send. That size is the full screen resolution used when the ereader is sleeping etc. KU then downscales the cover to match the required size for the library and grid views and saves the files.

If you're curious, thumbnail cache is in .kobo-images.

[davidfor]

Originally Posted by bigwoof

interesting! I always thought the Kobo was pulling the thumbnails out from the epub itself seeing how it slowly updates when you scroll through the list.

Well, it is. For some formats, including epubs and PDFs, it renders the first page and uses that. For kepubs, it looks for the image that has been marked as a cover and uses that.

and the epub thumbnail was resized appropriately before sending by the calibre kobo driver.

Which it can be. The KoboTouch driver generates and sends the three sizes needed by the device. And if you resend a book, it removes the existing covers so that they can be regenerated by the device, or the driver.

are the thumbnails actually stored on disk as well? can you just drop them and have the device pull it from the epub?

And if the device generates the covers, they are saved for later use. Look in the .kobo-images directory on the device.

[bigwoof]

Thanks for the excellent explanations @Davidfor and @Sherman..

so would a quick fix be to have an option to not receive the thumbnails separately from calibre in addition to the kepubs/epubs themselves?

Then just let the kobo device regenerate the thumbnail on its own time. This could be marked as a "enabling this will increase your memory usage. not recommended if you have a lot of books" option.

The other option would be to just receive them 1 by 1 and save them to disk immediately after processing. slow and steady..

[sherman]

Originally Posted by bigwoof

Thanks for the excellent explanations @Davidfor and @Sherman..

so would a quick fix be to have an option to not receive the thumbnails separately from calibre in addition to the kepubs/epubs themselves?

Then just let the kobo device regenerate the thumbnail on its own time. This could be marked as a "enabling this will increase your memory usage. not recommended if you have a lot of books" option.

The other option would be to just receive them 1 by 1 and save them to disk immediately after processing. slow and steady..

There is already an option in KU to not generate thumbnails, but looking at the code, I think it's broken for the 'none' option so...

[roger the rabbit]

@sherman: and much Kudos for UNCaGED.

Works perfectly for me saving the continual Micro USB wear & tear caused by cable unplugging.

Now Qi card wireless charging & Wifi book transfer extending the life of my Kobo.

(Had to junk an otherwise perfectly good phone for a broken Micro usb.)

As always MobileRead Forums are a great resource for the less technically endowed amongst us.

[sherman]

Hi all, I've released a beta version of v0.5.3. Releasing as beta due to limited testing. If no issues are uncovered, I will convert to a full release.

This release is primarily aimed at reducing memory usage, which hopefully allows for larger libraries to be used. YMMV on the library size howerver.

Note, books will now appear to take longer to send to the Kobo, as I've changed how the thumbnail images are created.

This release also fixes a bug that didn't disable thumbnail generation when asked.

As usual, this release can be found at releases page of the GitHub repository.