[Alpha] Kobo-UNCaGED - Calibre wireless connection

[uli]

Originally Posted by Carmelocotonto

Now All is OK except one question: books are downloaded on mnt/onboard. How can I send them to another dir?

When connected to Calibre look for the Device icon, Configure this Device, add a directory to the save template like:

Books/{authors:re(&.*,)}/{title} - {authors:re(&.*,)}

[Carmelocotonto]

Originally Posted by uli

When connected to Calibre look for the Device icon, Configure this Device, add a directory to the save template like:

Books/{authors:re(&.*,)}/{title} - {authors:re(&.*,)}

Thanks. This is I want.

[sherman]

Here is a new test version which removes the 'excluded formats' feature in favour of showing all the list of supported formats, and allowing you to toggle them at will.

EDIT: Test release deleted.

[MGlitch]

Originally Posted by sherman

Here is a new test version which removes the 'excluded formats' feature in favour of showing all the list of supported formats, and allowing you to toggle them at will.

Does this also include fixes for my issue, or have you not had the time for that yet?

[sherman]

Originally Posted by MGlitch

Does this also include fixes for my issue, or have you not had the time for that yet?

Yep. I think your issue was KU just being a bit too strict on its TCP deadline. @NiLuJe showed a similar behavior on the NOOP stuff (just without ever tripping the deadline).

I increased the timeout to 60 seconds (up from 15 seconds). I'm hoping this resolves it. Please test to confirm.

(I did it in the test version posted in #109. All test versions since then have included that fix)

[MGlitch]

Originally Posted by sherman

Yep. I think your issue was KU just being a bit too strict on its TCP deadline. @NiLuJe showed a similar behavior on the NOOP stuff (just without ever tripping the deadline).

I increased the timeout to 60 seconds (up from 15 seconds). I'm hoping this resolves it. Please test to confirm.

(I did it in the test version posted in #109. All test versions since then have included that fix)

As far as I can tell my issue is indeed resolved by the changes, I'd not tested before as format exclusion wasn't a big deal for me since I'm using calibre to control everything anyway. I say as far as I can tell since it's now been 8 minutes without it disconnecting and while I could just leave it for hours on end I doubt it'll disconnect given I was having issues getting it to get to a few minutes.

[sherman]

Originally Posted by MGlitch

As far as I can tell my issue is indeed resolved by the changes, I'd not tested before as format exclusion wasn't a big deal for me since I'm using calibre to control everything anyway. I say as far as I can tell since it's now been 8 minutes without it disconnecting and while I could just leave it for hours on end I doubt it'll disconnect given I was having issues getting it to get to a few minutes.

Thanks for the feedback. Good to hear your problem appears to be resolved.

[Carmelocotonto]

I have also tried the latest version and all went well. I llike this program very much.
One question: could it be the frontend more big?. Font bigger, etc...
I have got a kobo clara.
Thanks

[sherman]

Originally Posted by Carmelocotonto

I have also tried the latest version and all went well. I llike this program very much.
One question: could it be the frontend more big?. Font bigger, etc...
I have got a kobo clara.
Thanks

I've made some tweaks to the styling to allow slightly larger font sizes. There's only so much I can do without breaking the layout though, especially on the 6 inch devices. Changes will be in v0.5.2 when it is released.

[sherman]

Ok, v0.5.2 has been released! This includes the following fixes and features:

• Pull in a couple of fixes from UNCaGED to resolve a metadata parsing issue, and relax the timeout duration.
• Add new "Supported Formats" configuration. It is now possible to de-select specific ebook formats so that they are not presented to Calibre as supported formats. Useful for those who don't want Calibre to send specific formats to their Kobo
• Some style tweaks to support slightly larger font sizes in the UI

Thanks to those who have reported issues, and helped resolve them.

[bigwoof]

Hi!

First, thank you for a very useful app! This is great!

I've been trying to get it to work on my setup (Libra H20 with about 3K books and latest fully updated calibre running on linux) and I've found the following errors.

1. The Kobo "go to sleep" timer can trigger while this app is running and that messes everything app. needing a forced reboot actually.

Solution: turn off the sleep timer

2. I can never get the sync to finish. Even while it is transfering files, it disconnects from calibre and Kobo says that transfers are done.

Solution: not sure. I looked in the code to see where the disconnection timers were and I could not find it. I recall a test version with a longer 60s timer but I could not find the patch for that in the git repo

Thanks! Excellent app btw. very eager to get it to work on my setup. Which is an odd case as I do have about 3K books on my Libra H2O.

[sherman]

Originally Posted by bigwoof

Hi!

First, thank you for a very useful app! This is great!

I've been trying to get it to work on my setup (Libra H20 with about 3K books and latest fully updated calibre running on linux) and I've found the following errors.

1. The Kobo "go to sleep" timer can trigger while this app is running and that messes everything app. needing a forced reboot actually.

Solution: turn off the sleep timer

2. I can never get the sync to finish. Even while it is transfering files, it disconnects from calibre and Kobo says that transfers are done.

Solution: not sure. I looked in the code to see where the disconnection timers were and I could not find it. I recall a test version with a longer 60s timer but I could not find the patch for that in the git repo

Thanks! Excellent app btw. very eager to get it to work on my setup. Which is an odd case as I do have about 3K books on my Libra H2O.

Thanks for the feedback. As noted in the readme, I've cautioned about large libraries, mainly because I haven't tested that myself, and I was a bit wary about how well it would work.

For 2. is this happening during the initial sync (where it's sending metadata to calibre)? Or when sending books to the device?

The timeout stuff is in the upstream UNCaGED library (the initial idea, is that there could be multiple clients using the same library).

If you want to experiment with the timeout stuff and you're comfortable with Go, you might want to fork UNCaGED, make the modifications, change the import path in Kobo-UNCaGED to point to your fork, and build a customized version of the zip file.

If you are using v0.5.2, you may wish to look at the log file at .adds/kobo-uncaged/ku.log to see what errors, if any, are occurring.

[bigwoof]

Hi Sherman!

Originally Posted by sherman

Thanks for the feedback. As noted in the readme, I've cautioned about large libraries, mainly because I haven't tested that myself, and I was a bit wary about how well it would work.

yes I saw the note about large libraries. which is why I mentioned this is an odd case :-)

Originally Posted by sherman

For 2. is this happening during the initial sync (where it's sending metadata to calibre)? Or when sending books to the device?

I did a few more tests today morning and there is nothing odd (that I can see) in the Kobo ku.log.

From what I can see, the issue is that Calibre disconnects (for some reason) even though Kobo UnCaged still thinks it is connected. and this happens before all the new books are transferred.

For example, my test had 8 new books in epub format. I had setup Kobo Uncaged to only take kepub. so calibre started 8 conversion jobs (one for each book) and then for some reason tried to upload 7 of them (not 8).

and this job crashed with this error while sending book 1.

[Errno 104] Connection reset by peer

Traceback (most recent call last):
File "calibre/gui2/device.py", line 89, in run
File "calibre/gui2/device.py", line 592, in _upload_books
File "calibre/devices/smart_device_app/driver.py", line 50, in _synchronizer
File "calibre/devices/smart_device_app/driver.py", line 1497, in upload_books
File "calibre/devices/smart_device_app/driver.py", line 732, in _put_file
File "calibre/devices/smart_device_app/driver.py", line 614, in _send_byte_string
ConnectionResetError: [Errno 104] Connection reset by peer

will try to debug more over the weekend with calibre in debug mode as well.

Thanks!

[bigwoof]

okay. this is very mysterious. I did some additional testing today.

all tests done with calibre in debug mode, and all sleep timers turned off on kobo libra h20. kobo has 3411 books. kobo and calibre server (running on a linux machine) are on the same local network connected via WiFi and a ethernet switch. Kobo UnCaged is set to Prefer Kepub and only kepub is selected as an acceptable format. so all the epubs need to be converted first.

run 1: debug enabled in kobo uncaged. turned it on. got to the "waiting" screen. waited quite a bit and then the kobo rebooted!! no connection made to calibre (running in debug mode)

run 2: debug enabled in uncaged. turned it on. got to the waiting screen and then it exited with a "all done" (forgot the actual wording) message. no connection made to calibre

run 3: turned off debug in uncaged and turned it on. got to waiting screen and then it actually connected (got to the next screen with the disconnect option). calibre shows all the books (a lot of them) being checked and the job spy said "reading book list". connection looks good from what the log says

SMART_DEV ( 454.01:450.304) run attempt to open device socket
SMART_DEV ( 455.77: 1.759) open
SMART_DEV ( 455.79: 0.000) _call_client GET_INITIALIZATION_INFO wfr True
SMART_DEV ( 455.84: 0.044) open Device can stream books True
SMART_DEV ( 455.85: 0.000) open Device can stream metadata True
SMART_DEV ( 455.86: 0.000) open Device can receive book binary True
SMART_DEV ( 455.87: 0.000) open Device can delete multiple books True
SMART_DEV ( 455.88: 0.000) open Device can use cached metadata True
SMART_DEV ( 455.89: 0.000) open Cache uses lpaths True
SMART_DEV ( 455.90: 0.000) open Can send OK to sendbook True
SMART_DEV ( 455.91: 0.000) open Can accept library info True
SMART_DEV ( 455.92: 0.000) open Will ask for update books False
SMART_DEV ( 455.94: 0.000) open Will set temp mark when syncing read False
SMART_DEV ( 455.95: 0.000) open Client device kind
SMART_DEV ( 455.96: 0.000) open Client device name Kobo Libra H2O
SMART_DEV ( 455.97: 0.000) open Client app name Kobo UNCaGED
SMART_DEV ( 455.98: 0.000) open App version #: 391
SMART_DEV ( 455.99: 0.000) open max_book_packet_len 4096
SMART_DEV ( 456.00: 0.000) open Device wants UUID file names False
SMART_DEV ( 456.04: 0.030) open selected formats ['kepub']
SMART_DEV ( 456.05: 0.000) open extension path lengths {'kepub': 38}
SMART_DEV ( 456.06: 0.000) open cover height 1680
SMART_DEV ( 456.07: 0.000) open Device is_read sync col None
SMART_DEV ( 456.08: 0.000) open Device is_read_date sync col None
SMART_DEV ( 456.09: 0.000) card_prefix

- After a few minutes of reading the books, it decides to convert and send 77 books

READING LIST: Sync Now - force_sync= False
READING LIST: Device found to sync to: Kobo Libra H2O efb49d4c-7d70-4e5e-912b-b81773b6d14d
READING LIST: Syncing 77 books in 'Send to Kobo' to: efb49d4c-7d70-4e5e-912b-b81773b6d14d (location:None)

- It starts jobs for converting all the books and sending them 1 by 1 to the Kobo once converted.

- it sends about 3 books but then on the 4th book, *Calibre* crashes. just dies dead. and there is nothing I can see in the logs to say why. UnCaged just pops up a message saying "All done".

These are the last 10 lines in the calibre debug log. as far as I can tell, they look fine and look the same as previous files that were sent successfully

No details available.
SMART_DEV (1246.27: 0.038) set_progress_reporter
Job: 90 Upload one book to the device: The Multiverse Project: Warhammer 40,000 started
SMART_DEV (1246.47: 0.000) set_plugboards
Device using plugboard kepub SMART_DEVICE_APP None
Setting metadata in: The Multiverse Project: Warhammer 40,000 at: /tmp/calibre_5.8.1_tmp_4fij358g/iz7_mxaycaltmpfmt.kepub
2021-01-01 11:08:35 [DEBUG] KEPUBMetadataWriter::set_metadata - self.__class__=<class 'calibre_plugins.kepubmdwriter.metadata.writer.KEP UBMetadataWriter'>
2021-01-01 11:08:35 [DEBUG] KEPUBMetadataWriter::set_metadata - covers=[<Element {http://www.idpf.org/2007/opf}meta at 0x7f8c47e8acc0>]
2021-01-01 11:08:35 [DEBUG] KEPUBMetadataWriter::set_metadata - cover_id=cover
2021-01-01 11:08:35 [DEBUG] KEPUBMetadataWriter::set_metadata - found cover

I checked the ku.log on the Kobo as well. nothing in there except the expected "Resizing covers" messages.

Not sure how to continue debugging this to be honest..

Ideas?

[sherman]

Having never tried with such a large number of books, It's a bit hard to figure out what exactly is going on. I would find KU logs more useful than Calibre logs however.

I wonder if you're running into a memory consumption issue? At startup, KU builds a map/list of all books on device, first by the .metada.calibre file (if it exists), and then by reading all the entries in the Kobo SQLite database, then reconciling the two. This is all done in memory.

It would be interesting to see how much memory KU consumes. If you can SSH/telnet to your Kobo and run 'top' while running KU should give an idea. Sort by memory in top by pressing 'm'. Note, in busybox, it looks like actual memory usage is (%VSZ/100) * VSZ

EDIT: scratch the top idea, doesn't seem to work properly. You can use 'ps -o rss,args | grep /mnt/onboard/.adds/kobo-uncaged/bin/ku' to get a snapshot though.