Normal for kobo to take forever on "processing content" ?

[like00111]

So just got my kobo aura h2o today, I want to say that I like it... if I could use it to its full potential.

One problem I am having is transfering my 8500 books from calibre to the kobos 32 gb micro sd card. I left for about 7 hours and left my kobo transfering all the books from calibre. Once I came back I saw the screen was at 31% processed content. Looked at calibre and saw that it finished transfering the books in 42 mins.

I ended up unplugging and restarting my kobo thinking it was frozen, saw that 2100 books made it to the kobo and within a few moments the kobo begin "processing content" at 6%. its hard to tell if it is just being slow or frozen because the percentage goes up so slowly.

Is this normal for such a massive amount of books? I did 45 books at first just to test out my kobo and that seemed to process very quickly.

help? suggestions?

[darryl]

Just leave it alone until it is finished. If you think it is freezing try splitting the send jobs into smaller numbers of books. It is not abnormal for the device to take a long time to add so many books. So long as the device has enough storage this should not present a problem.

[like00111]

Originally Posted by darryl

Just leave it alone until it is finished. If you think it is freezing try splitting the send jobs into smaller numbers of books. It is not abnormal for the device to take a long time to add so many books. So long as the device has enough storage this should not present a problem.

The device doesn't have to be connected anymore at this point does it? I'm afraid I might corrupt something one way or another by unplugging it and restarting it during the process.

[Canuck_in_Japan]

8500 books? My Kobo takes a couple of minutes if I put 10-15 books on at once. I couldn't imagine almost 10,000!

[like00111]

Originally Posted by Canuck_in_Japan

8500 books? My Kobo takes a couple of minutes if I put 10-15 books on at once. I couldn't imagine almost 10,000!

Yeah... It is quite a bit I guess. Just scared I may have messed up some stuff with the restart that I did. Other wise I'm looking at around 12 hours for the whole process to finish. That is if my math was correct, seems like every percent takes 8 mins.

[davidfor]

Yes, 8500 books in one go would be at least 10 hours. Processing books is high CPU and disk I/O, chances are it would run out of battery first. So, leave it plugged into power.

[Gabriel.Voyager]

I've a problem in a bad converted kepub and my H2O remain for hour at the 6%
Maybe in the 8500 books there is one book that kobo can't processed.
Try to upload 200-300 books, then another 200-300 and find the bad formatted book that give you a problem.

If stay for hour at the 6% there is a problem by my experience.

sorry for my bad english

[chrisridd]

Does anyone know what the Kobo actually does when it is "Processing" new books?

Pulling metadata out of an epub/kepub's OPF file should not be an expensive operation.

[davidfor]

Originally Posted by chrisridd

Does anyone know what the Kobo actually does when it is "Processing" new books?

Pulling metadata out of an epub/kepub's OPF file should not be an expensive operation.

Between the time you disconnect and the home screen is displayed, it:

1. Scans the database for any books missing on the disk or whose size has changed. For both, it removes all details of the book from the database
2. Scans the disk for new books not in the database
3. For each new or changed book, extract the metadata from the book. For books formats with a ToC, extracts the ToC.
4. Add the books and their metadata plus the ToC to the database

I think the first two steps are why the processing screen always starts at 6%. The third step is what takes the time as the percent complete increments. I'm pretty sure the last step happens after the books have been processed. But, older firmware did this while the books were being processed.

The third step is obviously the most intensive. For epubs and kepubs, it has to open a zip file to read the OPF and NCX files. It will take time, but I agree it seems to take more time than I would expect. There might be something else happening, but I don't know what it would be.

[chrisridd]

"Disk" I/O is probably pretty bad on the devices, so steps 1 & 2 might not be blindingly quick. I'd hope step 3 wasn't extracting each book to a temporary directory or something like that.

Without attaching something like strace to the system while it is doing this processing it is going to be hard working out what the device is doing.

[frostschutz]

It extracts all books. That alone takes considerable amount of time.

And then for each book it creates a TON of database entries (about ~100 entries for each book, like an entry for each entry of the books table of contents). And while sqlite is a nice database solution for embedded devices, it's also slow as pots on the Kobo.

The process is anything but efficient - and entirely superfluous if there was an option to use folder navigation instead. Unfortunately that's only possible with alternative software such as KoReader.

[PeterT]

Nonsense to the superfluous nature of the processing. it's also used for items like displaying the table of contents of books.

[chrisridd]

The TOC seems pretty superfluous unless you're actually reading a book. Why would the ereader need to scan them ahead of time?

Have you seen the ereader "extract all books", @frostschutz? It is very easy to open a zip archive in memory and read the contents of arbitrary files (all disk reads), which is all that Kobo needs to do to get metadata. It does not need to unzip each book to disk (lots of reads and writes).

[frostschutz]

Originally Posted by chrisridd

unzip each book to disk

Not what I meant. To get at a ZIP's contents you have to extract it, even in memory. It's work for the CPU so it takes time. Of course the EPUB format is also at fault for this; it should have some visible metadata section that does not involve processing the zip entirely. As for separate metadata files I'm not sure if the Kobo would use those (but I doubt it).

My Kobo's database is ~23MB, without ToC entries it drops down to ~2MB. Unfortunately the Kobo notices when you try to open a book (while everything else looks completely normal).

[davidfor]

Originally Posted by chrisridd

The TOC seems pretty superfluous unless you're actually reading a book. Why would the ereader need to scan them ahead of time?

Strictly it doesn't need it. It is at least partly a historical thing. The early firmware on the original Kobo, and possibly the WiFi, handled the books differently. Extracting the ToC made sense for that. Now, it probably doesn't make as much sense, but rewriting code that works is always a very low priority.

For kepubs, the ToC entries in the database do contain extra information. That is generated the first time the book is opened.