Normal for kobo to take forever on "processing content" ?

[DNSB]

Originally Posted by eschwartz

That and the squillions of Kindle owners who own three books.

Not all that uncommon with any ereader. Those who want to carry a significant subset of the Library of Congress with them are quite rare. In my experience, the vast majority of ereader owners I've met average less than 12 books per year counting purchases and library loans.

Originally Posted by eschwartz

i.e. you said the words "around here". Highly unwise, since MobileRead is in no way representative of Amazon or Kobo's userbase or concerns.

Actually, by around here, I meant in my physical area and not the MobileRead virtual community. There are a fair number of users of both Kobo and Kindle ereaders along with a few less common devices. For my sins, I seem to be regarded as someone to ask about ereaders so I get a fair amount of input about various ereader models. "Which do you think is the best ereader?" (It's not being sold yet...) being the most common followed by "How do I get ebooks from the local library to my Kindle?" (Around here, you don't without significant error and help from Apprentice Alf).

BTW, the co-worker I mentioned who strips covers and other unneeded (in his opinion) data to shrink his .mobi files? We tried adding 142 books to his Kindle PW2 and it was faster but not blazingly faster than a Kobo Aura HD. The mobi and epub versions of the books were downloaded from Gutenberg with the .mobi files loaded into Calibre on his laptop and .epub files loaded into Calibre on my laptop. We attempted to click on Smart Eject in Calibre at the same time and then waited for processing to finish. Both ereaders already had quite a few ebooks added which may be a factor in the time taken to process new ebooks. I did take a second pass through Calibre to add the series info but didn't count that time since the processing time for adding that information is negligible.

The books added were a mix of Jerome K. Jerome, P. G. Wodehouse, Stanley R. Matthews, Victor Appleton (I and II) and O. A. Kline for those who might be interested.

[shalym]

Originally Posted by DNSB

Actually, by around here, I meant in my physical area and not the MobileRead virtual community. There are a fair number of users of both Kobo and Kindle ereaders along with a few less common devices. For my sins, I seem to be regarded as someone to ask about ereaders so I get a fair amount of input about various ereader models. "Which do you think is the best ereader?" (It's not being sold yet) being the most common followed by "How do I get ebooks from the library to my Kindle?" (Around here, you don't.).

BTW, the co-worker I mentioned who strips covers and other unneeded (in his opinion) data to shrink his .mobi files? We tried adding 142 books to his Kindle PW2 and it was faster but not blazingly faster than a Kobo Aura HD. The mobi and epub versions of the books were downloaded from Gutenberg with the .mobi files loaded into Calibre on his laptop and .epub files loaded into Calibre on my laptop. We attempted to click on Smart Eject in Calibre at the same time and then waited for processing to finish. Both ereaders already had quite a few ebooks added which may be factor in the time taken to process new ebooks.

How did you know when the Kindle was done processing? I just plugged in my PW2 and added 86 books using Calibre, and when I ejected it I was immediately able to start reading--immediately. I then added those same 86 books to my Aura HD and when I ejected I had to spend the next 2.5 minutes staring at a black screen that informed me that it was processing the content. That's the difference between the Kindle and the Kobo. The Kindle is able to process the new content in the background while it is also being used to read a book. Why can't Kobo do that?

Shari

[xor_]

Originally Posted by shalym

For me, the biggest difference between the Kobo and the Kindle in terms of processing time is that the Kobo is unusable while new books are being processed, and the Kindle is able to be used.

This. The way Kobo handles the book processing is a terrible design decision. I haven't seen this level of annoyance with any other ebook reader. Your device can become completely unusable for an unpredictable time period. If you're lucky it's done after a few hours. Unless it chokes on some bad formatting and doesn't finish at all. Without even giving you an error message. It's insane. And if you forget to unmount your device and the database gets corrupted, it does it all again.

As a user I don't care how the software does its processing. But chances are if I have my device in my hands and upload some books, I want to read. Not watch a black screen with a progress percentage.

And I actually like my Kobo devices, but this madness on the software side is a major reason why I rarely bother using nickel anymore.

Edit: I recently upgraded from the Glo to the Glo HD. But I did it despite the Kobo software. If it wasn't for koreader, I would have bought a Kindle.

[davidfor]

Originally Posted by chrisridd

Is there any way to investigate deeper to profile what's happening (at the broad level as we don't have source code) when the Kobo says it is "Processing"? Maybe if someone could telnet to the ereader and run strace?

Yes, telnet and strace should be possible. Hopefully someone will be interested enough to try.

[frostschutz]

I did a strace, it just wasn't very conclusive. I can only see the parts where it works on the database, I don't see it actually reading the files. Maybe that's hidden in another process somehow?

[pid  1216] lstat64("/mnt/onboard/NEWBOOKS/10384.epub", {st_mode=S_IFREG|0755, st_size=415487, ...}) = 0
[pid  1216] clock_gettime(CLOCK_MONOTONIC, {467, 645462820}) = 0
[pid  1216] fcntl64(25, F_SETLK64, {type=F_RDLCK, whence=SEEK_SET, start=1073741824, len=1}, 0x36d0dfd8) = 0
[pid  1216] fcntl64(25, F_SETLK64, {type=F_RDLCK, whence=SEEK_SET, start=1073741826, len=510}, 0x36d0dfd8) = 0
[pid  1216] fcntl64(25, F_SETLK64, {type=F_UNLCK, whence=SEEK_SET, start=1073741824, len=1}, 0x36d0dfd8) = 0
[pid  1216] access("/mnt/onboard/.kobo/KoboReader.sqlite-journal", F_OK) = -1 ENOENT (No such file or directory)
[pid  1216] fstat64(25, {st_mode=S_IFREG|0755, st_size=23808000, ...}) = 0
[pid  1216] _llseek(25, 24, [24], SEEK_SET) = 0
[pid  1216] read(25, "\0\0\n\264\0\0Z\322\0\0\0\0\0\0\0\0", 16) = 16
[pid  1216] fstat64(25, {st_mode=S_IFREG|0755, st_size=23808000, ...}) = 0
[pid  1216] access("/mnt/onboard/.kobo/KoboReader.sqlite-wal", F_OK) = -1 ENOENT (No such file or directory)
[pid  1216] fstat64(25, {st_mode=S_IFREG|0755, st_size=23808000, ...}) = 0
[pid  1216] fcntl64(25, F_SETLK64, {type=F_WRLCK, whence=SEEK_SET, start=1073741825, len=1}, 0x36d0e068) = 0
[pid  1216] clock_gettime(CLOCK_MONOTONIC, {467, 648696820}) = 0
[pid  1216] clock_gettime(CLOCK_MONOTONIC, {467, 648814195}) = 0
[pid  1216] clock_gettime(CLOCK_MONOTONIC, {467, 649031945}) = 0
[pid  1216] writev(26, [{"l\1\0\1\32\0\0\0\177\0\0\0\177\0\0\0\1\1o\0\25\0\0\0/org/fre"..., 144}, {"\25\0\0\0com.kobo.adobe.Parser\0", 26}], 2) = 170
[pid  1216] gettimeofday({1434473391, 282060}, NULL) = 0
[pid  1216] poll([{fd=26, events=POLLIN}], 1, 25000) = 1 ([{fd=26, revents=POLLIN}])
[pid  1216] read(26, "l\2\1\1\0\0\0\0N\0\0\0005\0\0\0\6\1s\0\4\0\0\0:1.0\0\0\0\0"..., 2048) = 161
[pid  1216] read(26, 0x302ec648, 2048)  = -1 EAGAIN (Resource temporarily unavailable)
[pid  1216] writev(26, [{"l\1\1\1\210\0\0\0\200\0\0\0\177\0\0\0\1\1o\0\25\0\0\0/org/fre"..., 144}, {"\203\0\0\0type='signal',sender='org.fr"..., 136}], 2) = 280
[pid  1216] writev(26, [{"l\1\0\1,\0\0\0\201\0\0\0w\0\0\0\1\1o\0\1\0\0\0/\0\0\0\0\0\0\0"..., 136}, {"'\0\0\0file:///mnt/onboard/NEWBOOKS"..., 44}], 2) = 180
[pid  1216] gettimeofday({1434473391, 287120}, NULL) = 0
[pid  1216] poll([{fd=26, events=POLLIN}], 1, 25000) = 1 ([{fd=26, revents=POLLIN}])
[pid  1216] read(26, "l\2\1\1\0\0\0\0P\0\0\0005\0\0\0\6\1s\0\4\0\0\0:1.0\0\0\0\0"..., 2048) = 72
[pid  1216] read(26, 0x302ec648, 2048)  = -1 EAGAIN (Resource temporarily unavailable)
[pid  1216] gettimeofday({1434473391, 288660}, NULL) = 0
[pid  1216] poll([{fd=26, events=POLLIN}], 1, 24999) = 1 ([{fd=26, revents=POLLIN}])
[pid  1216] read(26, "l\2\1\1\311\1\0\0005\0\0\0005\0\0\0\6\1s\0\4\0\0\0:1.0\0\0\0\0"..., 2048) = 529
[pid  1216] read(26, 0x302ec648, 2048)  = -1 EAGAIN (Resource temporarily unavailable)
[pid  1216] writev(26, [{"l\1\0\1,\0\0\0\202\0\0\0w\0\0\0\1\1o\0\1\0\0\0/\0\0\0\0\0\0\0"..., 136}, {"'\0\0\0file:///mnt/onboard/NEWBOOKS"..., 44}], 2) = 180
[pid  1216] gettimeofday({1434473391, 358273}, NULL) = 0
[pid  1216] poll([{fd=26, events=POLLIN}], 1, 25000) = 1 ([{fd=26, revents=POLLIN}])
[pid  1216] read(26, "l\2\1\1\230\4\0\0006\0\0\0005\0\0\0\6\1s\0\4\0\0\0:1.0\0\0\0\0"..., 2048) = 1248
[pid  1216] read(26, 0x302ec648, 2048)  = -1 EAGAIN (Resource temporarily unavailable)
[pid  1216] clock_gettime(CLOCK_MONOTONIC, {467, 731162445}) = 0
[pid  1216] _llseek(25, 10564608, [10564608], SEEK_SET) = 0
[pid  1216] read(25, "\2\0\0\0\3\1\350\0\0\0*Z\1\350\2\203\3D\1\200\0\265\0\32\0\32\0\0)\273\201\25"..., 1024) = 1024
[pid  1216] _llseek(25, 11040768, [11040768], SEEK_SET) = 0
[pid  1216] read(25, "\2\0\0\0\6\0L\0\0\0*R\3\200\3\33\2\255\1\354\1\33\0L\0\0\0\0\0\0\0\0"..., 1024) = 1024
[pid  1216] _llseek(25, 11039744, [11039744], SEEK_SET) = 0
[pid  1216] read(25, "\2\0\0\0\5\0007\0\0\0*%\2}\1\273\0\371\0007\3?\0\0\0\0\0\0\0\0\0\0"..., 1024) = 1024
[pid  1216] _llseek(25, 11046912, [11046912], SEEK_SET) = 0
[pid  1216] read(25, "\n\3C\0\10\0A\0\3\0\2\252\0021\1\265\0019\0\275\0A\3\204\0\0\0\0\0\0\0\0"..., 1024) = 1024
[pid  1216] stat64("/mnt/onboard/.kobo/KoboReader.sqlite", {st_mode=S_IFREG|0755, st_size=23808000, ...}) = 0
[pid  1216] open("/mnt/onboard/.kobo/KoboReader.sqlite-journal", O_RDWR|O_CREAT|O_LARGEFILE|O_CLOEXEC, 0755) = 27
[pid  1216] fstat64(27, {st_mode=S_IFREG|0755, st_size=0, ...}) = 0
[pid  1216] geteuid32()                 = 0
[pid  1216] fchown32(27, 0, 0)          = 0
[pid  1216] fstat64(27, {st_mode=S_IFREG|0755, st_size=0, ...}) = 0
[pid  1216] _llseek(27, 0, [0], SEEK_SET) = 0
[pid  1216] write(27, "\0\0\0\0\0\0\0\0\0\0\0\0\311\334\370\v\0\0Z\322\0\0\2\0\0\0\4\0\0\0\0\0"..., 512) = 512
[pid  1216] _llseek(27, 512, [512], SEEK_SET) = 0
[pid  1216] write(27, "\0\0*%", 4)      = 4
...
[pid  1216] _llseek(28, 6168, [6168], SEEK_SET) = 0
[pid  1216] write(28, "\0\0=\10", 4)    = 4
[pid  1216] _llseek(28, 6172, [6172], SEEK_SET) = 0
[pid  1216] write(28, "\n\0\0\0\t\0J\0\0\223\1P\2\r\2V\2\237\2\350\0J\0031\3\207\0\0\0\0\0\0"..., 1024) = 1024
[pid  1216] _llseek(28, 7196, [7196], SEEK_SET) = 0
[pid  1216] write(28, "\0\0=\t", 4)     = 4
[pid  1216] _llseek(28, 7200, [7200], SEEK_SET) = 0
[pid  1216] write(28, "\n\0\0\0\6\1\30\0\1\30\1\224\2\20\2\214\3\10\3\204\0#\3\204\0\0\0\0\0\0\0\0"..., 1024) = 1024
[pid  1216] _llseek(28, 8224, [8224], SEEK_SET) = 0
[pid  1216] write(28, "\0\0Z\334", 4)   = 4
[pid  1216] _llseek(28, 8228, [8228], SEEK_SET) = 0
[pid  1216] write(28, "\r\0\0\0\4\0029\0\3\217\3\35\2\253\0029\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024) = 1024
[pid  1216] clock_gettime(CLOCK_MONOTONIC, {467, 919661820}) = 0
[pid  1216] clock_gettime(CLOCK_MONOTONIC, {467, 919792070}) = 0
[pid  1216] stat64("/mnt/onboard/NEWBOOKS/10384.epub", {st_mode=S_IFREG|0755, st_size=415487, ...}) = 0

[eschwartz]

Originally Posted by davidfor

The Kobo is opening the book to get the title and author. As the rest of the metadata is either in the OPF or comes from the file system, it isn't going to add much time to it.

Surely it requires time to write to the database, and makes the database bigger, etc. -- even if there isn't any additional penalty for extracting the OPF?

Word count, or even page count, would be better as I sometimes choose a book based on size (due to mood, opportunity to read etc). But, I have a fair idea of the book length based on the file size using my experience with the books and the known sources.

I have found filesize to be wildly erratic. Not only does filesize not have any good reason to be a good indicator of book length, it conforms to my expectations by not being a good indicator.

A format of epub/kepub, PDF, CBR/CBZ, text etc does tell me something - fiction, text book or other info, comic book and "why do I have plain text on the device?"

I can get all that from the title and cover image.

As the sideloaded books are not directly linked to a shop somewhere, using that method is useless. Plus about half of my read is done somewhere without connectivity, so it wouldn't help. And if I want to look at the jacket, that means I have to open the book which marks it on the Kobo as open. I can unmark it, but, that's extra work. And from what I can see, the reason that calibre metadata jackets exist is because so few of the ereaders support displaying this information separately. There is no way I would buy one that didn't.

Very true. Amazon wants you to buy books from their store.
I don't really look at summaries, but if I did I would be disappointed on most devices, as you pointed out.

I'll beg to disagree. Embedding the series info in the title is a hack done because the device can't display it separately.

Perhaps that is why I agreed with you...

The Kobo's generate the cover images as needed. It does waste some space by having different sizes for different places, but it works fine. Or calibre can generate them during the send.

The simple thing here is that the point of the metadata being in the book is that applications can use it. And hopefully in a way that is useful to the user. I am interested in seeing most of the metadata above. I do look at it when choosing a book. So, the time taken in building this is useful to me. If Kobo speed it up, then great. If they add a bit more info, just as good (I'd love to see tags and filter based on them).

And I still cannot understand why half that metadata needs exist outside of an archival database.

Originally Posted by Terisa de morgan

@eschwartz, sorry, yes, that's the info I was talking about. Not interesting for you, the main reason I won't have a Kindle unless they change library management or I have no other option. Different people, different choices, but it keeps on being more metadata extraction in Kobo than Kindle.

I guess I understand where you are coming from... partly. I still think it seems to be extracting more metadata than is actually needed.

[eschwartz]

Originally Posted by shalym

For me, the biggest difference between the Kobo and the Kindle in terms of processing time is that the Kobo is unusable while new books are being processed, and the Kindle is able to be used.

In other words, I really don't care how long the Kindle takes, as I can still read a book while it is processing the new content.

Shari

Exactly.

[eschwartz]

Originally Posted by DNSB

Not all that uncommon with any ereader. Those who want to carry a significant subset of the Library of Congress with them are quite rare. In my experience, the vast majority of ereader owners I've met average less than 12 books per year counting purchases and library loans.



Actually, by around here, I meant in my physical area and not the MobileRead virtual community. There are a fair number of users of both Kobo and Kindle ereaders along with a few less common devices. For my sins, I seem to be regarded as someone to ask about ereaders so I get a fair amount of input about various ereader models. "Which do you think is the best ereader?" (It's not being sold yet...) being the most common followed by "How do I get ebooks from the local library to my Kindle?" (Around here, you don't without significant error and help from Apprentice Alf).

Apologies for the assumption.

However I would like to still point out that most people don't really read all that much -- and they are Amazon's target audience.
Kobo's target audience is: people who want to fiddle with their device. Few of those will be the I-read-12-books-a-year crowd.

I don't believe your physical area is representative.

[davidfor]

Originally Posted by eschwartz

Surely it requires time to write to the database, and makes the database bigger, etc. -- even if there isn't any additional penalty for extracting the OPF?

But, as I want the information available, I don't consider it a penalty.

I have found filesize to be wildly erratic. Not only does filesize not have any good reason to be a good indicator of book length, it conforms to my expectations by not being a good indicator.

That is both true and not true. For a set of well constructed epubs with no cover that have all been compress using the same method, file size is a reasonable indicator of book length. That is part of why I included "my experience" and "known sources" in my statement.

I can get all that from the title and cover image.

That of course depends on the title and the cover. Of course, I can also sort the library list by file type. If I decided to read a comic book, I can use that to group them all together and choose.

Very true. Amazon wants you to buy books from their store.
I don't really look at summaries, but if I did I would be disappointed on most devices, as you pointed out.


Perhaps that is why I agreed with you...

Well, you said "fair substitute". I didn't agree that series info in the title is that.

And I still cannot understand why half that metadata needs exist outside of an archival database.

I guess I understand where you are coming from... partly. I still think it seems to be extracting more metadata than is actually needed.

This all really comes down to "why not?". And is it useful to some people? The metadata is there in the books. As the device can see it, why not expose it to the user. And as all the metadata is coming from one file inside the book, the extra time is minimal. The database size will increase, but only the synopsis is going to be really noticeable.

Looking back at the list of metadata that I can see on a Kobo ereader, the only bits that I don't use when choosing a book is the ISBN and language (I only read/speak English, though my wife has her opinion on that). The publisher is the least useful, but it does tell me source if the book isn't a traditionally published book, so I sometimes look at it. And of course the cover might have some influence. My decision is almost always made using the ereader and away from calibre and even a network connection. That means I have to rely on my memory or what the ereader shows me.

[eschwartz]

Originally Posted by davidfor

Well, you said "fair substitute". I didn't agree that series info in the title is that.

Fair, not great. Not fair as in reasonable. Sorry for the confusion.

[chrisridd]

The strace is interesting.

It would be useful to see it with timestamps on each line, but I think you're right that there is some other process doing the parsing - you can see (I think) nickel apparently handing over the filename and required parser on fd 26, and then waiting to read results back.

[pid  1216] writev(26, [{"l\1\0\1\32\0\0\0\177\0\0\0\177\0\0\0\1\1o\0\25\0\0\0/org/fre"..., 144}, {"\25\0\0\0com.kobo.adobe.Parser\0", 26}], 2) = 170
[pid  1216] gettimeofday({1434473391, 282060}, NULL) = 0
[pid  1216] poll([{fd=26, events=POLLIN}], 1, 25000) = 1 ([{fd=26, revents=POLLIN}])
[pid  1216] read(26, "l\2\1\1\0\0\0\0N\0\0\0005\0\0\0\6\1s\0\4\0\0\0:1.0\0\0\0\0"..., 2048) = 161
[pid  1216] read(26, 0x302ec648, 2048)  = -1 EAGAIN (Resource temporarily unavailable)
[pid  1216] writev(26, [{"l\1\1\1\210\0\0\0\200\0\0\0\177\0\0\0\1\1o\0\25\0\0\0/org/fre"..., 144}, {"\203\0\0\0type='signal',sender='org.fr"..., 136}], 2) = 280
[pid  1216] writev(26, [{"l\1\0\1,\0\0\0\201\0\0\0w\0\0\0\1\1o\0\1\0\0\0/\0\0\0\0\0\0\0"..., 136}, {"'\0\0\0file:///mnt/onboard/NEWBOOKS"..., 44}], 2) = 180
[pid  1216] gettimeofday({1434473391, 287120}, NULL) = 0
[pid  1216] poll([{fd=26, events=POLLIN}], 1, 25000) = 1 ([{fd=26, revents=POLLIN}])
[pid  1216] read(26, "l\2\1\1\0\0\0\0P\0\0\0005\0\0\0\6\1s\0\4\0\0\0:1.0\0\0\0\0"..., 2048) = 72

A little after all this the KoboReader.sqlite database is opened and updated. It seems to use a journal file.

One of the strings starts "free" - I wonder if that is "freedesktop", and I wonder if a Qt mechanism is being used.

[Terisa de morgan]

Of course, once you have obtained the file, you unzip it, get the opf file and parse it. But it should be the standard way of working. You have to analyze the tags for getting the metadata.

[tshering]

Originally Posted by frostschutz

I did a strace, it just wasn't very conclusive. I can only see the parts where it works on the database, I don't see it actually reading the files. Maybe that's hidden in another process somehow?

Would calling strace with -f or -ff be helpful?

[frostschutz]

That _was_ strace with -ff. I'll see if I can find the mystery process. It did not show up in a pstree that I triggered while it was updating books, but it might be a shortlived process and I tried to nab it at the wrong time.

Apparently ZIP allows direct file access so if the metadata is the last file in the ZIP (which is the case for toc in Calibre generated EPUBs) it does not have to extract all files that come before it. Whether the system actually uses that kind of optimization, or parses all HTML as well, I don't know.

As for making it a background process it should be possible because other things already happen in the background (if you sync you get a thin 5 pixel progress bar at the top of the screen).

[meeera]

Originally Posted by xor_

This. The way Kobo handles the book processing is a terrible design decision. I haven't seen this level of annoyance with any other ebook reader. Your device can become completely unusable for an unpredictable time period. If you're lucky it's done after a few hours.

Processing time taking many hours affects a vanishingly small group of users. And for those three or so whom it does affect, this is surely an extremely rare event - just how many of those 8500 books is the user going to get through in the next year and need to replace?

To me, and to the overwhelming majority of users, this is simply a non-issue. That forty or so seconds out of my life while my Kobo processes a handful more books isn't actually going to make a big impact on my day. If I have to do a factory reset or something and want to add back a hundred or so, I can simply do it at a time when I have other stuff to do for a while. And I am a very heavy reader compared to average.