Normal for kobo to take forever on "processing content" ?

[frostschutz]

Originally Posted by meeera

To me, and to the overwhelming majority of users, this is simply a non-issue.

Certainly, it's some high level criticism. Kobo readers are great after all, or we wouldn't be using them.

But I don't think it's a non-issue. It's a first impression thing and first impressions are important. You buy your first Kobo, you copy your book collection on it, and you can't read. If it's your first Kobo device (like, coming from Kindle) you don't know it will do this and once it starts it's too late.

And that's exactly what seems to have happened to the original poster of this thread ("just got my kobo aura h2o today -- 8500 books -- left for about 7 hours -- 31% done"). It's ridiculous how slow it is.

Often it's the little details that make a great user experience and this is one area where the Kobo could use some improvement.

[Terisa de morgan]

I totally agree with meera I don't know a lot of users who add so many books in a row, and taking half minute or one minute for adding the three/four books I usually add, it's not a problem, so I don't either see the big gain for Kobo. And yes, I'm a heavy reader and I connect my ereader very frequently to the computer.

[Terisa de morgan]

How many users send 8.500 books the first time? I know none in real life, and only a few in forum, so it's a minor point.

[meeera]

Originally Posted by Terisa de morgan

How many users send 8.500 books the first time? I know none in real life, and only a few in forum, so it's a minor point.

I find it hard to imagine any complete newbie just going ahead and adding 8500 books (can you even do that to a non-modded reader?). This is really not a first impressions issue.

What I personally like about the Kobo's approach is that once the processing is done, it's done, and you can see what's going on. There's no temporary reader slowdown while processing a big batch of books in the background.

[frostschutz]

As for processing times, experiment directly on the Kobo:

# echo 3 > /proc/sys/vm/drop_caches
# time sh -c 'find /mnt/ -name "*.epub" -exec unzip -p {} content.opf \; | grep dc:title'

This extracts content.opf from all EPUB (I only have EPUB, nothing else) and prints the book title. And it does so in an inefficient way (find and unzip is magnitude slower than what can be done in a C program).

For me this takes 16 seconds for ~400 books. Directly on the Kobo itself.

I then emptied my KoboReader.sqlite to make the Kobo reprocess the same books.

After 1 minute it was at 8% complete.
After 2 minutes, 13%.
After 3 minutes, 15%.

I stopped tracking the progress at that point. It's still not done while I'm typing this post (37% after several more minutes).

If I can grab the essential data required to display a list of books in 16 seconds on the Kobo shell using an inefficient method, it's hard to understand why the Kobo is so fricking slow about doing the same thing.

[frostschutz]

Originally Posted by meeera

I find it hard to imagine any complete newbie just going ahead and adding 8500 books (can you even do that to a non-modded reader?).

The H2O (and pretty much every Kobo except Glo HD?) does have an external SD card slot so you have to expect people popping in an SD card filled to the brim with books. Blocking the device for hours is not optimal.

[chrisridd]

I think you may need to do an lsof during "Processing" to see if you can find the other end of the pipe, if busybox supports that. Or can you look through /proc if you're really quick?

Using the widely available unzip library code, a caller can open a zip/epub file, and then step through the zip file's internal structure in order to access any given file. This could be more efficient if the OPF is at the end of the epub (zip files are walked from the end of file backwards!), but even if it is not it is a few seek()s and read()s away at most. You absolutely don't have to unpack the entire file.

So your testing with find and unzip really is worst case. Yet still faster than Kobo!!

Have you tried updating an sqlite database? I think you said earlier in this thread that sqlite perf was pretty bad on the Kobo.

[frostschutz]

adobehost process

[pid  4473] clock_gettime(CLOCK_MONOTONIC, {268, 951787739}) = 0
[pid  4473] clock_gettime(CLOCK_MONOTONIC, {268, 951946114}) = 0
[pid  4473] clock_gettime(CLOCK_MONOTONIC, {268, 952128614}) = 0
[pid  4473] pselect6(9, [3 8], [], [], {3, 49000000}, {NULL, 8}) = 1 (in [3], left {3, 48982375})
[pid  4473] read(3, "\1\0\0\0\0\0\0\0", 8) = 8
[pid  4473] clock_gettime(CLOCK_MONOTONIC, {268, 952781364}) = 0
[pid  4473] clock_gettime(CLOCK_MONOTONIC, {268, 953240364}) = 0
[pid  4473] clock_gettime(CLOCK_MONOTONIC, {268, 953393989}) = 0
[pid  4473] clock_gettime(CLOCK_MONOTONIC, {268, 953521989}) = 0
[pid  4473] pselect6(9, [3 8], [], [], {3, 47000000}, {NULL, 8}) = 1 (in [8], left {2, 535180625})
[pid  4473] read(8, "l\1\0\1R\0\0\0\247\0\0\0\205\0\0\0\1\1o\0\1\0\0\0/\0\0\0\0\0\0\0"..., 2048) = 234
[pid  4473] read(8, 0x3da88, 2048)      = -1 EAGAIN (Resource temporarily unavailable)
[pid  4473] write(3, "\1\0\0\0\0\0\0\0", 8) = 8
[pid  4473] clock_gettime(CLOCK_MONOTONIC, {269, 466764989}) = 0
[pid  4473] open("/mnt/onboard/ZEFIX/005 - Durch das Land der Skipetaren - Karl May.epub", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 11
[pid  4473] fcntl64(11, F_SETFD, FD_CLOEXEC) = 0
[pid  4473] fstat64(11, {st_mode=S_IFREG|0755, st_size=400056, ...}) = 0
[pid  4473] fstat64(11, {st_mode=S_IFREG|0755, st_size=400056, ...}) = 0
[pid  4473] read(11, "PK\3\4\24\0\26\10\0\0)=l:oa\253,\24\0\0\0\24\0\0\0\10\0\0\0mi"..., 16384) = 16384
[pid  4473] close(11)                   = 0
[pid  4473] stat64("/mnt/onboard/ZEFIX/005 - Durch das Land der Skipetaren - Karl May.epub", {st_mode=S_IFREG|0755, st_size=400056, ...}) = 0
[pid  4473] open("/mnt/onboard/ZEFIX/005 - Durch das Land der Skipetaren - Karl May.epub", O_RDONLY) = 11
[pid  4473] fstat64(11, {st_mode=S_IFREG|0755, st_size=400056, ...}) = 0
[pid  4473] mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2fb54000
[pid  4473] fstat64(11, {st_mode=S_IFREG|0755, st_size=400056, ...}) = 0
[pid  4473] _llseek(11, 397312, [397312], SEEK_SET) = 0
[pid  4473] read(11, "\364^-\320\23p<R\370\332\335\3336\307u\17\17\326z\351\375\354N\366\207G\320\341_\4\335\277"..., 2744) = 2744
[pid  4473] mmap2(NULL, 400056, PROT_READ, MAP_PRIVATE, 11, 0) = 0x2fb55000
[pid  4473] munmap(0x2fb55000, 400056)  = 0
[pid  4473] close(11)                   = 0
[pid  4473] munmap(0x2fb54000, 4096)    = 0
[pid  4473] writev(8, [{"l\2\1\1\t\4\0\0J\0\0\0#\0\0\0\6\1s\0\4\0\0\0:1.0\0\0\0\0"..., 56}, {"\1\4\0\0\0\0\0\0\n\0\0\0DC.creator\0\1s\0\0\0\10\0\0\0"..., 1033}], 2) = 1089
[pid  4473] clock_gettime(CLOCK_MONOTONIC, {269, 521851864}) = 0
[pid  4473] clock_gettime(CLOCK_MONOTONIC, {269, 522000989}) = 0
[pid  4473] clock_gettime(CLOCK_MONOTONIC, {269, 522150739}) = 0
[pid  4473] pselect6(9, [3 8], [], [], {2, 479000000}, {NULL, 8}) = 1 (in [3], left {2, 478981875})
[pid  4473] read(3, "\1\0\0\0\0\0\0\0", 8) = 8
[pid  4473] clock_gettime(CLOCK_MONOTONIC, {269, 522738239}) = 0
[pid  4473] clock_gettime(CLOCK_MONOTONIC, {269, 522881114}) = 0
[pid  4473] clock_gettime(CLOCK_MONOTONIC, {269, 523004239}) = 0
[pid  4473] clock_gettime(CLOCK_MONOTONIC, {269, 523124364}) = 0
[pid  4473] pselect6(9, [3 8], [], [], {2, 478000000}, {NULL, 8}) = 1 (in [8], left {2, 476355625})
[pid  4473] read(8, "l\1\0\1R\0\0\0\250\0\0\0\205\0\0\0\1\1o\0\1\0\0\0/\0\0\0\0\0\0\0"..., 2048) = 234
[pid  4473] read(8, 0x3da88, 2048)      = -1 EAGAIN (Resource temporarily unavailable)
[pid  4473] write(3, "\1\0\0\0\0\0\0\0", 8) = 8
[pid  4473] clock_gettime(CLOCK_MONOTONIC, {269, 526128239}) = 0
[pid  4473] open("/mnt/onboard/ZEFIX/005 - Durch das Land der Skipetaren - Karl May.epub", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 11
[pid  4473] fcntl64(11, F_SETFD, FD_CLOEXEC) = 0
[pid  4473] fstat64(11, {st_mode=S_IFREG|0755, st_size=400056, ...}) = 0
[pid  4473] fstat64(11, {st_mode=S_IFREG|0755, st_size=400056, ...}) = 0
[pid  4473] read(11, "PK\3\4\24\0\26\10\0\0)=l:oa\253,\24\0\0\0\24\0\0\0\10\0\0\0mi"..., 16384) = 16384
[pid  4473] close(11)                   = 0
[pid  4473] stat64("/mnt/onboard/ZEFIX/005 - Durch das Land der Skipetaren - Karl May.epub", {st_mode=S_IFREG|0755, st_size=400056, ...}) = 0
[pid  4473] open("/mnt/onboard/ZEFIX/005 - Durch das Land der Skipetaren - Karl May.epub", O_RDONLY) = 11
[pid  4473] fstat64(11, {st_mode=S_IFREG|0755, st_size=400056, ...}) = 0
[pid  4473] mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2fb54000
[pid  4473] fstat64(11, {st_mode=S_IFREG|0755, st_size=400056, ...}) = 0
[pid  4473] _llseek(11, 397312, [397312], SEEK_SET) = 0
[pid  4473] read(11, "\364^-\320\23p<R\370\332\335\3336\307u\17\17\326z\351\375\354N\366\207G\320\341_\4\335\277"..., 2744) = 2744
[pid  4473] mmap2(NULL, 400056, PROT_READ, MAP_PRIVATE, 11, 0) = 0x2fb55000
[pid  4473] semop(1376264, {{0, -1, SEM_UNDO}}, 1) = 0
[pid  4473] semop(1376264, {{0, 1, SEM_UNDO}}, 1) = 0
[pid  4473] semop(1376264, {{0, -1, SEM_UNDO}}, 1) = 0
[pid  4473] semop(1376264, {{0, 1, SEM_UNDO}}, 1) = 0
[pid  4473] semop(1376264, {{0, -1, SEM_UNDO}}, 1) = 0
[pid  4473] semop(1376264, {{0, 1, SEM_UNDO}}, 1) = 0
[pid  4473] semop(1376264, {{0, -1, SEM_UNDO}}, 1) = 0
[pid  4473] semop(1376264, {{0, 1, SEM_UNDO}}, 1) = 0
[pid  4473] semop(1376264, {{0, -1, SEM_UNDO}}, 1) = 0
[pid  4473] semop(1376264, {{0, 1, SEM_UNDO}}, 1) = 0
[pid  4473] semop(1376264, {{0, -1, SEM_UNDO}}, 1) = 0
[pid  4473] semop(1376264, {{0, 1, SEM_UNDO}}, 1) = 0
[pid  4473] gettimeofday({1434551829, 838316}, NULL) = 0
[pid  4473] send(10, "<15>Jun 17 16:37:09 adobehost: v"..., 255, MSG_NOSIGNAL) = 255
[pid  4473] semop(1376264, {{0, -1, SEM_UNDO}}, 1) = 0
[pid  4473] semop(1376264, {{0, 1, SEM_UNDO}}, 1) = 0
[pid  4473] munmap(0x2fb55000, 400056)  = 0
[pid  4473] close(11)                   = 0
[pid  4473] munmap(0x2fb54000, 4096)    = 0
[pid  4473] writev(8, [{"l\2\1\1m\10\0\0K\0\0\0$\0\0\0\6\1s\0\4\0\0\0:1.0\0\0\0\0"..., 56}, {"i\10\0\0\35\1\0\0\2\0\0\0id\0\1s\0\0\0\214\0\0\0file:///"..., 2157}], 2) = 2213
[pid  4473] clock_gettime(CLOCK_MONOTONIC, {270, 338781864}) = 0
[pid  4473] clock_gettime(CLOCK_MONOTONIC, {270, 338937239}) = 0
[pid  4473] clock_gettime(CLOCK_MONOTONIC, {270, 339094239}) = 0
[pid  4473] pselect6(9, [3 8], [], [], {1, 662000000}, {NULL, 8}) = 1 (in [3], left {1, 661981625})
[pid  4473] read(3, "\1\0\0\0\0\0\0\0", 8) = 8
[pid  4473] clock_gettime(CLOCK_MONOTONIC, {270, 339771739}) = 0
[pid  4473] clock_gettime(CLOCK_MONOTONIC, {270, 339920739}) = 0
[pid  4473] clock_gettime(CLOCK_MONOTONIC, {270, 340166239}) = 0
[pid  4473] clock_gettime(CLOCK_MONOTONIC, {270, 340325239}) = 0
[pid  4473] pselect6(9, [3 8], [], [], {1, 661000000}, {NULL, 8}) = 1 (in [8], left {1, 201296625})
[pid  4473] read(8, "l\1\0\1>\0\0\0\252\0\0\0\205\0\0\0\1\1o\0\1\0\0\0/\0\0\0\0\0\0\0"..., 2048) = 214
[pid  4473] read(8, 0x3da88, 2048)      = -1 EAGAIN (Resource temporarily unavailable)
[pid  4473] write(3, "\1\0\0\0\0\0\0\0", 8) = 8
[pid  4473] clock_gettime(CLOCK_MONOTONIC, {270, 801502864}) = 0
[pid  4473] open("/mnt/onboard/ZEFIX/006 - Der Schut - Karl May.epub", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 11
[pid  4473] fcntl64(11, F_SETFD, FD_CLOEXEC) = 0
[pid  4473] fstat64(11, {st_mode=S_IFREG|0755, st_size=444137, ...}) = 0
[pid  4473] fstat64(11, {st_mode=S_IFREG|0755, st_size=444137, ...}) = 0
[pid  4473] read(11, "PK\3\4\24\0\26\10\0\0&?l:oa\253,\24\0\0\0\24\0\0\0\10\0\0\0mi"..., 16384) = 16384
[pid  4473] close(11)                   = 0

It opens the file and memory maps it, what happens then is anyone's guess.

It makes sense, I guess, to let some Adobe software handle it. I tend to forget there are such things as copy protected books.

[davidfor]

Originally Posted by frostschutz

As for processing times, experiment directly on the Kobo:

# echo 3 > /proc/sys/vm/drop_caches
# time sh -c 'find /mnt/ -name "*.epub" -exec unzip -p {} content.opf \; | grep dc:title'

This extracts content.opf from all EPUB (I only have EPUB, nothing else) and prints the book title. And it does so in an inefficient way (find and unzip is magnitude slower than what can be done in a C program).

For me this takes 16 seconds for ~400 books. Directly on the Kobo itself.

I then emptied my KoboReader.sqlite to make the Kobo reprocess the same books.

After 1 minute it was at 8% complete.
After 2 minutes, 13%.
After 3 minutes, 15%.

I stopped tracking the progress at that point. It's still not done while I'm typing this post (37% after several more minutes).

If I can grab the essential data required to display a list of books in 16 seconds on the Kobo shell using an inefficient method, it's hard to understand why the Kobo is so fricking slow about doing the same thing.

Except that you have only done part of it. It is extracting other metadata from the OPF. Which means it has to parse the the OPF properly rather than just a simple grep. Then it is extracting the NCX file and parsing the contents and building structures based on that. How much more should that take, I don't know. I do agree that the Kobo is taking a lot longer than I would expect. I just don't know if this is because the code is inefficient, or because it is doing something that we don't know about.

All this has prompted me to do a test I've been meaning to do for a while. I wanted to compare the time taken and the size of the file and ToC. So, I generated an epub that was simply 4503 copies of "<h2>Chapter nnnn</h2>". Then I had calibre generate a ToC for that. I dropped that on my Glo HD and timed the processing. Timing from when the light turned on when ejecting to when the home screen showed, took about 55 seconds. But, the book wouldn't open and when I looked in the database, there were no ToC entries.

I changed the file to only 1000 ToC entries and put 10 copies on the Glo HD. The file is 62KB They took about 88 seconds to process. This did open OK - 72 pages of ToC!

Then I took a 560KB book I had that was made with nearly all the text in single section. It as simple text cover and about 30 chapters. I removed all but to of the chapters from the ToC. I put ten copies of this on the Glo HD. That took 22 seconds to process.

That isn't a completely conclusive, but it agrees with what I have been thinking: the size of the ToC affected the processing time more than the size of the book.

[frostschutz]

Originally Posted by davidfor

Except that you have only done part of it.

Certainly. But it's the part that matters most (if you put a random filename 123456789.epub on your reader, it should show the real title "Adventures of Huckleberry Finn" instead). If you don't do at least that much, you have a file browser (which I wouldn't mind really, my books have sensible names after all so I could do without any kind of database).

[xor_]

Originally Posted by meeera

To me, and to the overwhelming majority of users, this is simply a non-issue.

People who pontificate other people's issues to be non-issues, while arrogating to speak for an overwhelming majority, always make me scratch my head.

Of course someone adding 8,500 books is an extreme case. But FWIW, I've seen annoyingly long delays with just a few tens of books. I let it run over night, and in the morning it still wasn't finished.
Stuck at the same percentage of processing. Without even an error message. That's completely unacceptable to me.

I found out that it's probably choking on some bad formatting, and have adjusted by running everything through calibre first. I learned to jump through hoops because of a weakness in Kobo's software design. I shouldn't have to do that. And, notabene, that was the same set of books I had used before on a Nook and a Kindle without any comparable delay.

Edit:
And, as someone else has said, to me whatever Kobo is trying to achieve by "processing" a database is a non-issue. It serves no purpose for me. Just load the file, give me some basic formatting, and I'm happy. So this whole book processing, while having an annoying downside, isn't even a tradeoff that has an upside.

[eschwartz]

Originally Posted by meeera

I find it hard to imagine any complete newbie just going ahead and adding 8500 books (can you even do that to a non-modded reader?). This is really not a first impressions issue.

What I personally like about the Kobo's approach is that once the processing is done, it's done, and you can see what's going on. There's no temporary reader slowdown while processing a big batch of books in the background.

Oh, you mean just like the Kindle, except that there still isn't a slowdown?

Okay, granted, the Kindle doesn't show the progress. But I'd rather read than stare at a progressbar, and it sounds like I am not the only one who is startled by the idea of a progressbar being desirable.

The one downside I can see is that if a book jams the indexer (that happens on Kindles too, @xor_ ) then you won't have any idea except by looking at the number of books you now have.
Of course, on either the Kindle or the Kobo you will have to remove all the non-indexed books and add a few at a time to figure out which book is problematic.

[DNSB]

Originally Posted by eschwartz

The one downside I can see is that if a book jams the indexer (that happens on Kindles too, @xor_ ) then you won't have any idea except by looking at the number of books you now have.
Of course, on either the Kindle or the Kobo you will have to remove all the non-indexed books and add a few at a time to figure out which book is problematic.

Personal opinion is that finding those malformed books is ever so much fun though Flightcrew and epubcheck make it a lot easier for epubs. OTOH, I sometimes wonder if anyone other than myself uses Flightcrew and epubcheck on a regular basis to check incoming ebooks.

[eschwartz]

Oh yes, wonderful fun.
I am happy to say I've never had to deal with that myself.

[DNSB]

Originally Posted by xor_

I found out that it's probably choking on some bad formatting, and have adjusted by running everything through calibre first. I learned to jump through hoops because of a weakness in Kobo's software design. I shouldn't have to do that. And, notabene, that was the same set of books I had used before on a Nook and a Kindle without any comparable delay.

While I'd agree that Kobo's error checking is about as useful as a screendoor on a submarine, any commercially produced epubs should pass Flightcrew and/or epubcheck without major errors. For non-commercial epubs, I'll allow a bit of leniency though many of them are more standards compliant than many of the commercial epubs. I have not found an epub that validated with no major errors and caused my Kobo reader to hang during processing.

As for the vast pool of pirated ebooks converted from ghod alone knows what format to epub? I tend to feel that if I am going to read a book, I should be paying the author whether directly or through the library system. I will admit that I have no qualms about writing authors to comment on a book whether in a positive or negative fashion. I've learned to be careful with phrasing -- suggesting to an author that, in my not so humble opinion, their literary output indicates they should be pursuing another line of endeavour such as dog catcher seems to be considered as a vicious attack.

"This is not a novel to be tossed aside lightly. It should be thrown with great force"

Often attributed to Dorothy Parker. Dorothy Parker occupies a similar position as Oscar Wilde in that much of what is attributed to them originated with others.