[Device Plugin] Kobo Touch Extended Driver - Page 170

davidfor · 06-13-2021, 07:55 AM

@inthemood: How many internal file does the book have? The driver is starting a thread for each file to make the changes. And the error is when a new thread is started. A book with a lot of files could cause a problem. Is the edition of the book somewhere I can download it? It looks like it is Dante's Divine Comedy, but, it would be better to use the same version.

inthemood · 06-13-2021, 08:07 AM

Thank you David. I attached the epub. I have problems with a lot of other large epubs. It is a problem I didn't have previously though. I'm having to reload my books in a new reader as my previous one was stolen. When I uploaded them to my old er-eader (same brand - same model as the one I am using now: kobo forma), there were no such issues.

compurandom · 06-13-2021, 11:45 AM

Quote:

Originally Posted by inthemood

[...]
crashes randomly (it might work the next time I try)
[...]
RuntimeError: Failed to process Divina Commedia by Dante Alighieri: can't start new thread

Looks like you are running out of memory or some other resource.

inthemood · 06-13-2021, 02:57 PM

Quote:

Originally Posted by compurandom

Looks like you are running out of memory or some other resource.

Yes. When I try to upload a "large" book onto my e-reader, the calibre process gradually uses up almost all the cpu (of which I have quite a bit) and then Calibre crashes. That doesn't happen every time, but I'm left with books I can't load onto the reader. First converting them from epub to kepub seems to make things easier…

JSWolf · 06-13-2021, 03:11 PM

Quote:

Originally Posted by inthemood

Yes. When I try to upload a "large" book onto my e-reader, the calibre process gradually uses up almost all the cpu (of which I have quite a bit) and then Calibre crashes. That doesn't happen every time, but I'm left with books I can't load onto the reader. First converting them from epub to kepub seems to make things easier…

Are you running v5.x 64 or32-bit?

inthemood · 06-13-2021, 03:17 PM

Quote:

Originally Posted by JSWolf

Are you running v5.x 64 or32-bit?

Calibre 5.2.1 - 64 bit mac os x mojave.

davidfor · 06-14-2021, 01:56 AM

Quote:

Originally Posted by inthemood

Thank you David. I attached the epub. I have problems with a lot of other large epubs. It is a problem I didn't have previously though. I'm having to reload my books in a new reader as my previous one was stolen. When I uploaded them to my old er-eader (same brand - same model as the one I am using now: kobo forma), there were no such issues.

As @compurandom said, it is almost definitely a resource issue. Looking at at the book, it has 4435 xhtml files in the manifest that need to be handled. The driver starts a thread for each file and then waits for them to finish. In this case it is probably starting to many for the OS to handle at one time. It probably should be managing this a bit better. I'll try and look at that.

The structure of the book is the issue. I am not a fan of how it was constructed. The reason for the large number of files is that the text for each footnote is in a separate file. And there are a lot of footnotes. Merging the footnotes into a single file for each verse should solve this and should still be usable.

Joliet Jake · 06-23-2021, 11:02 AM

Does KTE use a different method for transferring covers than Kobo Touch, Kobo Utilities, etc? Or does Kobo ignore metadata covers for Kepubs but not Epubs?

I used to use Kobo Touch and, at the time, I would use metadata to "attach" covers to my sideloaded books. Then I would transfer them to my Kobo. The correct covers would appear.

Well recently my Kobo "went on strike". I had to update it, log out, log in, etc, and the problems generally went away. However, I had a lot of problems with covers. Some books didn't have covers, and some had "old" covers. The books weren't using the covers added using metadata.

I made sure to use the exact same settings in KTE as in Kobo Touch.

So, I converted those Epubs into Epubs and that solved the problem, as the first page was replaced by the metadata cover (or even a new first page was added), except there's a few books with DRM issues. I can't edit them and so they keep the old covers (or lack of covers). One set of books were definitely purchased without DRM but I'm getting a DRM error on them anyway. (This does not prevent me from reading the books either on Calibre or the Kobo. I just cannot edit them.) Those books actually have the covers inside, but on page 3 (page 1 is blank, and that's what shows up on the Kobo).

In addition, I cannot find "Polish". I am not sure if that's a separate plugin but I think it's native to Calibre, so I'm probably just looking at the wrong menu. (I tried modifying the menu but I didn't know which menu needed editing so I gave up.) I don't know if Polish would make any difference.

Edit: Davidfor answered this question on another post of mine.

av_mtl · 06-25-2021, 01:44 PM

I got my Kobo Elipsa (firmware v4.28.17826) today and I noticed that the stylus doesn't work with kepubs converted by the extended driver. Ran a quick test case just to see and kepubify has the same issue. If the same books are uploaded in their original epub format, it works fine.

I know this is super new and unforeseeable, but wondering if anyone has ideas for a workaround? I love the reading stats capabilities of kepubs so I'm a bit sad to switch back to epubs on this device.

av_mtl · 06-25-2021, 01:54 PM

*Clarification: the stylus's markup functionality doesn't work, for navigation it does.

jgoguen · 07-04-2021, 12:10 PM

Quote:

Originally Posted by av_mtl

I got my Kobo Elipsa (firmware v4.28.17826) today and I noticed that the stylus doesn't work with kepubs converted by the extended driver. Ran a quick test case just to see and kepubify has the same issue. If the same books are uploaded in their original epub format, it works fine.

I know this is super new and unforeseeable, but wondering if anyone has ideas for a workaround? I love the reading stats capabilities of kepubs so I'm a bit sad to switch back to epubs on this device.

I don't have an Elipsa and I'm not likely to be getting one anytime soon so I doubt I'll be able to debug this. If anyone has an extra Elipsa they want to donate or has an extra $450 US they don't know what to do with I'll take it and see what I can come up with, but no promises. The Elipsa doesn't give me anything I really want and even if it did it's too expensive for me to consider.

JSWolf · 07-04-2021, 01:00 PM

Quote:

Originally Posted by av_mtl

I got my Kobo Elipsa (firmware v4.28.17826) today and I noticed that the stylus doesn't work with kepubs converted by the extended driver. Ran a quick test case just to see and kepubify has the same issue. If the same books are uploaded in their original epub format, it works fine.

I know this is super new and unforeseeable, but wondering if anyone has ideas for a workaround? I love the reading stats capabilities of kepubs so I'm a bit sad to switch back to epubs on this device.

There is a patch posted at the KePub GitHub to allow you to mark up side loaded KePub.

https://github.com/pgaskin/kepubify/issues/66

davidfor · 07-04-2021, 11:41 PM

There have been a few reports recently of errors when sending very large books using this driver. The error occurs in the code that is starting the threads to process each internal file. The released version starts one thread for each internal file. This is causing problems with very large books when using the 32bit version of calibre. I am not sure of the exact size of book that will cause a problem, but, the test book I am using has over 5000 internal files.

The attached version of the plugin solves this problem by only starting a smaller number of threads that do the work. At the moment, it is starting up to 100 threads. That should cover most books, but, also prevent larger books from causing issues. There might be some performance impact, but, my testing is inconclusive. The comparisons have been done while other processes have been running that would have affected the time taken. With the large file mentioned, both the released and beta versions are taking between 5 and half and 6 minutes.

Please test this and report any problems. Most of the testing is with the 64bit current release of calibre. I have done very quick tests with the portable versions of 3.48 and 4.23 and it appears to be working with them.

This problem also affects the output plugin for conversions. I have attached a version of this as well if anyone needs it.

anacreon · 07-05-2021, 06:51 AM

Quote:

Originally Posted by davidfor

There have been a few reports recently of errors when sending very large books using this driver. The error occurs in the code that is starting the threads to process each internal file. The released version starts one thread for each internal file. This is causing problems with very large books when using the 32bit version of calibre. I am not sure of the exact size of book that will cause a problem, but, the test book I am using has over 5000 internal files.

The attached version of the plugin solves this problem by only starting a smaller number of threads that do the work. At the moment, it is starting up to 100 threads. That should cover most books, but, also prevent larger books from causing issues. There might be some performance impact, but, my testing is inconclusive. The comparisons have been done while other processes have been running that would have affected the time taken. With the large file mentioned, both the released and beta versions are taking between 5 and half and 6 minutes.

Please test this and report any problems. Most of the testing is with the 64bit current release of calibre. I have done very quick tests with the portable versions of 3.48 and 4.23 and it appears to be working with them.

This problem also affects the output plugin for conversions. I have attached a version of this as well if anyone needs it.

I have tested this with Elipsa / calibre portable 5.22. Since this morning I was in a hurry, and made an error that necessitated a rough exit, I had to restore the last data base saved, 03 July late.
First I sent the three "ordinary" epubs I had sent to Elipsa yesterday 4 July, and it worked. I ejected/reconnected, then sent the metadata and reading positions, checked the DB (ok) and saved it. They all landed as kepubs on the Elipsa. I then disabled the "silent" check box on KoboTouch Extended, to have the eventual error message to send you.

I tried a test copy of one of the 11 books which necessitated a preliminary conversion with Calibre 64bit 5.22.1, "Dictionnaire historique de la langue française", 1114 html files, 20.10 Mo, and it didn't work. Here is the error message:

Spoiler:

I am of course at your disposal for further tests whenever I have the time - not that much presently since I go on holidays Thursday 8, but I'll take my Surface, Elipsa and Libra and can continue from the 11th.

davidfor · 07-05-2021, 07:26 AM

Quote:

Originally Posted by anacreon

I have tested this with Elipsa / calibre portable 5.22. Since this morning I was in a hurry, and made an error that necessitated a rough exit, I had to restore the last data base saved, 03 July late.
First I sent the three "ordinary" epubs I had sent to Elipsa yesterday 4 July, and it worked. I ejected/reconnected, then sent the metadata and reading positions, checked the DB (ok) and saved it. They all landed as kepubs on the Elipsa. I then disabled the "silent" check box on KoboTouch Extended, to have the eventual error message to send you.

I tried a test copy of one of the 11 books which necessitated a preliminary conversion with Calibre 64bit 5.22.1, "Dictionnaire historique de la langue française", 1114 html files, 20.10 Mo, and it didn't work. Here is the error message:

Spoiler:

I am of course at your disposal for further tests whenever I have the time - not that much presently since I go on holidays Thursday 8, but I'll take my Surface, Elipsa and Libra and can continue from the 11th.

Can you run calibre in debug mode and post or PM the log? I need to see what is happening to understand why it isn't working for you. Also, is the book available to see? And how much RAM does the Surface have? Anything else running at the time to consume resources?

06-23-2021, 11:02 AM	#2543
Joliet Jake Zealot Posts: 111 Karma: 3098438 Join Date: Feb 2021 Location: Canada Device: Kobo Sage	Does KTE use a different method for transferring covers than Kobo Touch, Kobo Utilities, etc? Or does Kobo ignore metadata covers for Kepubs but not Epubs? I used to use Kobo Touch and, at the time, I would use metadata to "attach" covers to my sideloaded books. Then I would transfer them to my Kobo. The correct covers would appear. Well recently my Kobo "went on strike". I had to update it, log out, log in, etc, and the problems generally went away. However, I had a lot of problems with covers. Some books didn't have covers, and some had "old" covers. The books weren't using the covers added using metadata. I made sure to use the exact same settings in KTE as in Kobo Touch. So, I converted those Epubs into Epubs and that solved the problem, as the first page was replaced by the metadata cover (or even a new first page was added), except there's a few books with DRM issues. I can't edit them and so they keep the old covers (or lack of covers). One set of books were definitely purchased without DRM but I'm getting a DRM error on them anyway. (This does not prevent me from reading the books either on Calibre or the Kobo. I just cannot edit them.) Those books actually have the covers inside, but on page 3 (page 1 is blank, and that's what shows up on the Kobo). In addition, I cannot find "Polish". I am not sure if that's a separate plugin but I think it's native to Calibre, so I'm probably just looking at the wrong menu. (I tried modifying the menu but I didn't know which menu needed editing so I gave up.) I don't know if Polish would make any difference. Edit: Davidfor answered this question on another post of mine. Last edited by Joliet Jake; 06-23-2021 at 12:47 PM.

07-04-2021, 11:41 PM	#2548
davidfor Grand Sorcerer Posts: 24,905 Karma: 47303824 Join Date: Jul 2011 Location: Sydney, Australia Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos	Beta - Fix issues with very large books There have been a few reports recently of errors when sending very large books using this driver. The error occurs in the code that is starting the threads to process each internal file. The released version starts one thread for each internal file. This is causing problems with very large books when using the 32bit version of calibre. I am not sure of the exact size of book that will cause a problem, but, the test book I am using has over 5000 internal files. The attached version of the plugin solves this problem by only starting a smaller number of threads that do the work. At the moment, it is starting up to 100 threads. That should cover most books, but, also prevent larger books from causing issues. There might be some performance impact, but, my testing is inconclusive. The comparisons have been done while other processes have been running that would have affected the time taken. With the large file mentioned, both the released and beta versions are taking between 5 and half and 6 minutes. Please test this and report any problems. Most of the testing is with the 64bit current release of calibre. I have done very quick tests with the portable versions of 3.48 and 4.23 and it appears to be working with them. This problem also affects the output plugin for conversions. I have attached a version of this as well if anyone needs it. Last edited by davidfor; 07-05-2021 at 09:36 PM. Reason: Removed beta as official versions have been released.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Kobo Device Driver Update	davidfor	Devices	284	05-24-2019 05:24 PM
[Device Plugin] Kindle 2, 3, 4, Touch Device Interface MBP Update Mod	cryzed	Plugins	7	10-28-2012 04:58 PM
[Device Interface Plugin] Update for Nook Color Driver	jmricker	Plugins	0	10-22-2011 10:11 AM
Touch Kobo Touch Extended 2 Year Warranty - Is it worth the $34.99?	EverC	Kobo Reader	11	08-31-2011 11:47 PM
Touch Kobo Touch Extended Warranty Program	SensualPoet	Kobo Reader	1	07-17-2011 04:08 AM

06-13-2021, 07:55 AM	#2536
davidfor Grand Sorcerer Posts: 24,905 Karma: 47303824 Join Date: Jul 2011 Location: Sydney, Australia Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos	@inthemood: How many internal file does the book have? The driver is starting a thread for each file to make the changes. And the error is when a new thread is started. A book with a lot of files could cause a problem. Is the edition of the book somewhere I can download it? It looks like it is Dante's Divine Comedy, but, it would be better to use the same version.

06-25-2021, 01:44 PM	#2544
av_mtl Junior Member Posts: 4 Karma: 10 Join Date: Jun 2021 Device: Kobo Aura H2O (1st gen), Kobo Forma, Kobo Elipsa	I got my Kobo Elipsa (firmware v4.28.17826) today and I noticed that the stylus doesn't work with kepubs converted by the extended driver. Ran a quick test case just to see and kepubify has the same issue. If the same books are uploaded in their original epub format, it works fine. I know this is super new and unforeseeable, but wondering if anyone has ideas for a workaround? I love the reading stats capabilities of kepubs so I'm a bit sad to switch back to epubs on this device.

06-25-2021, 01:54 PM	#2545
av_mtl Junior Member Posts: 4 Karma: 10 Join Date: Jun 2021 Device: Kobo Aura H2O (1st gen), Kobo Forma, Kobo Elipsa	*Clarification: the stylus's markup functionality doesn't work, for navigation it does.