Glo Glo stalls during scan of reloaded collection ?.

[leftright]

Kobo Glo v3.12. All epubs sideloaded.

I deleted all books in my Glo using the remove function in settings.
I transferred my entire collection from Calibre to Glo, transfer Ok.
I disconnected from PC and the Glo's scan feature started, but it stalled at 80%, I left it for two days and the progress bar didn't move past 80%, pin hole reset was required to restart the Glo, Zero books in it.

I connected Glo to Calibre, Calibre shows that books are in Glo.
I re-deleted all books via settings and proceeded to transfer books from Calibre to Glo piecemeal, after each small transfer I disconnected the Glo letting it scan, then I'd reconnect the Glo to Calibre for another transfer.
It took forever but ALL books were successfully scanned by the Glo.

1). Why would the glo hang during the scan of the entire collection and not during the piecemeal reload and rescan ?.

2). If it does encounter a problem book, why doesn't it skip that book and continue ?.

Suggestion:
If there are problem books that it stalls on, skip them and continue with the loading of subsequent books, then display a fault log at the end.

[booklover6]

No way would I wait two days for processing.

I've been playing around with my Aura, and now it's acting wonky. Once it was processing for about an hour. I doubt I would let it go longer than that.

[davidfor]

Originally Posted by leftright

Kobo Glo v3.12. All epubs sideloaded.

I deleted all books in my Glo using the remove function in settings.
I transferred my entire collection from Calibre to Glo, transfer Ok.
I disconnected from PC and the Glo's scan feature started, but it stalled at 80%, I left it for two days and the progress bar didn't move past 80%, pin hole reset was required to restart the Glo, Zero books in it.

I connected Glo to Calibre, Calibre shows that books are in Glo.
I re-deleted all books via settings and proceeded to transfer books from Calibre to Glo piecemeal, after each small transfer I disconnected the Glo letting it scan, then I'd reconnect the Glo to Calibre for another transfer.
It took forever but ALL books were successfully scanned by the Glo.

1). Why would the glo hang during the scan of the entire collection and not during the piecemeal reload and rescan ?.

2). If it does encounter a problem book, why doesn't it skip that book and continue ?.

Suggestion:
If there are problem books that it stalls on, skip them and continue with the loading of subsequent books, then display a fault log at the end.

How many books did you send? As they processed ok in small groups, the issue was probably the bulk lot. I think the maximum I have processed in one go is about 1500. That was after a logout and back in and took a couple of hours. My guess would be that as the number of books gets larger, any inefficiencies in the code get amplified. It's build a list of books and their details, if it needs to search that list as a new book is added, that get slower with each book. And if there are any code problems, such as a small memory leak per book, then when you hit several thousand books, they will build and cause a bigger problem. Remember, these are very low power devices with little RAM. Expecting them to process stuff as quickly and efficiently as your desktop PC is absurd.

And before you say "But they should fix these problems". Of course they should. But, they have to choose which problem to fix first. Do they fix a problem that only affects a small number of people doing an uncommon task (loading thousands of books in one go) that also has a reasonably easy workaround (load them in batches), or a problem that affects a lot of people? They also have to know about the problem. If no-one has reported a bug, they can't fix it.

[leftright]

Originally Posted by davidfor

How many books did you send?

848 side loaded epubs.

Is there any way of verifying a entire collection for errors which would cause a ereader to "stall", in calibre or Sigil ?, this would prevent issues when loading the collection to a lightweight ereader whose processing software is buggy.

It's exasperating how many OEM's push a product out the door which don't do the basics right, then expect the user to find work-around for the bugs. *sigh*.

If it weren't for excellent support forums, like this one, their product would fail miserably. Perhaps it's part of the business plan, "push the thing out the door asap, bugs and all, let the nice folks out there deal with support for free".

Modern Managers and Bean counters have no shame. No integrity. No work ethic.

[davidfor]

Originally Posted by leftright

848 side loaded epubs.

That number should have been processed with no problems.

Is there any way of verifying a entire collection for errors which would cause a ereader to "stall", in calibre or Sigil ?

Unfortunately, I have no idea what in the book causes this. When this happened a lot, I would ask people if they worked out the book that caused the problem to send it to me to look at. I rarely received a book and on the occasions that I did, couldn't find anything that seemed to cause it. Most times, when I loaded it, there were no problems. With the changes in the firmware, the problem has nearly disappeared. When I started here, there was a question about it most days. I can't think of the last time I saw one.

The problem is in either the OPF or NCX. I am reasonably sure that these are the only files that are looked at when the book is processed. Other than extracting the metadata, I don't know what it does with the OPF. For the NCX, it reads the TOC and puts that into the database. It doesn't check that what the ToC points exists anywhere in the book.

I am confident that any epub that has been opened and saved with Sigil or the calibre editor and neither of those showed any errors will work. I'm also sure any book that has been produced by a calibre conversion to epub will be OK.

[leftright]

For info;
Calibre Companion and FBreader process the same collection with no issues when transferred in bulk.

[PeterT]

Unfortunately, the fact other s/w handles it is irrelevant here.

[leftright]

Originally Posted by davidfor

a reasonably easy workaround (load them in batches),

Right there is a solution.
How difficult would it be to add a IF clause to the Kobo "OS" to auto scan a large collection in batches of its own accord. Why must the user figure out a workaround ?.

[leftright]

Originally Posted by davidfor

it reads the TOC and puts that into the database

I'm now going through each book in my Calibre collection to verify the integrity of the TOC and also do a "bug" check. I have come across four books whose TOC was absolutely atrocious, it took Calibre several seconds to load the book's TOC and there would be a pause of a few seconds each time I deleted a pointless link, I eventually lost patience and deleted the entire TOC, that also locked up the TOC edit window for 30+ seconds.

If Calibre struggles then I understand how the Glo "stalls".

Why does the Kobo "OS" scan the TOC when the book is transferred, why doesn't it load the TOC only if the book is opened ?, surely that makes more sense and would prohibit the stalling issue.

[davidfor]

Originally Posted by leftright

I'm now going through each book in my Calibre collection to verify the integrity of the TOC and also do a "bug" check. I have come across four books whose TOC was absolutely atrocious, it took Calibre several seconds to load the book's TOC and there would be a pause of a few seconds each time I deleted a pointless link, I eventually lost patience and deleted the entire TOC, that also locked up the TOC edit window for 30+ seconds.

If Calibre struggles then I understand how the Glo "stalls".

Yes, if calibre on a PC is having trouble, then the low-powered CPU in the ereader is going to get stuck. What was wrong with the ToCs?

Why does the Kobo "OS" scan the TOC when the book is transferred, why doesn't it load the TOC only if the book is opened ?, surely that makes more sense and would prohibit the stalling issue.

I believe it has something to do with how the older devices and firmware worked. With at least one of them, they unpacked the epubs when they were added to the device. Building the database entries while doing that probably made sense. There is a clear path of evolution in the down through the firmware, so the current versions do a lot of things the same as the older versions. It would seem to make sense to extract the ToC when the book is first opened, but, unless Kobo sees a real benefit in spending the time and resources on it, it isn't going to change.

[leftright]

Originally Posted by davidfor

What was wrong with the ToCs?

Most of the books TOC's had several Pointless, i.e. red dot entries, and two books had entire paragraphs as a single pointer entry, with red dots, tons of those entries. Calibre was struggling so much with Individual TOC entries that I just deleted the TOC's en mass, I didn't spend much time diving into the code to determine the faults, I'm no coder anyway, don't understand it.

[leftright]

Glo v3.12: 788 sideloaded epubs.

I scanned each book in Calibre for errors, corrected all TOC, CSS and HTML errors.
Transferred entire collection to Glo, unplugged Glo.
Glo scan stalled at 92%. pin reset. Zero books in Glo.
I removed all books from Glo, transferred books to Glo in two batches.
First batch of 500 books Glo scanned OK
Second batch scanned ok but Glo froze on home screen after scanning.
Pin reset.
Reloaded second batch, Glo scan ok and displayed home screen OK.

Seems to me that there is a bug in the frimware which doesn't like loading large collections of books.

[GeoffR]

Originally Posted by leftright

Seems to me that there is a bug in the frimware which doesn't like loading large collections of books.

A hardware fault is also a possibility, especially if it stalls at a different place each time.

The hours of heavy computation involved in processing a large quantity of books in one go is not the sort of thing these devices were designed for. They are much more suited to short bursts of activity followed by long pauses where the CPU sleeps waiting for input.