Normal for kobo to take forever on "processing content" ?

[eschwartz]

P.S. It is indeed accurate to say that Kindles will be "slow" when you have many many books on them -- if by "slow", you mean "slow at performing a full-library inside-the-book search".

This has been explained to you before, yet you insist on bringing up the claim that Kindles "slow down" when they have many books.

[Terisa de morgan]

@eschwartz, there's another question. You can't compare the metadata info Kobo extracts per book with Kindle info. And we're not talking about info for later search but for showing them at the library.

[eschwartz]

What info do you speak of?

[davidfor]

Originally Posted by eschwartz

Slow down == take longer per book.

Yes, it does take more time when there are a lot of books being processed. It reads the books in one step and then updates the database in another. The time for the reads should linear, but as the data structure grows it will have affect. And the writes to the database will be affected by the size of the database. And unless there is a reorg done somewhere along the way, it is just going to get worse when processing a lot of books in one go.

The difference isn't that noticeable with the current firmware. The last time I setup a new Kobo I did some tests and found that adding ten batches of 100 was pretty much the same as all 1000 in one go. And all the extra connects and disconnects meant the batches took longer. I've never tried to add as many as the OP, but, I would expect several hours of processing and the risk of running out of battery, so I would leave the device plugged in.

[davidfor]

Originally Posted by eschwartz

What info do you speak of?

The metadata that is extracted is:

- Title
- Author
- Synopsis
- Language
- File size
- Format
- ISBN (actually takes the first identifier)

As well as that, the library can show:

- Publisher
- Published date
- Series name and number

Kobo populate these for purchased books from their server, but they have to be populated by other methods for sideloaded books. And I have no idea why Kobo do not extract the publisher and date as it is standard metadata. The cover is not extracted and is generated the first time it is needed.

I have no idea how the above compares to what is on the Kindle.

[DNSB]

Originally Posted by eschwartz

Kobo users tend to be the ones who care more about having lots of books, so a disproportionate number of users would care.

Odd. Around here, I find as many Kindle as Kobo owners who have the desire to have massive libraries on their devices. The only real difference is that the Kobo owners, for the most part, have the option of easily added external storage so it's easier to have massive numbers of books.

[DNSB]

Originally Posted by eschwartz

I've never tried adding 8500 books at once -- it would probably require more than the 3.5 GB available.

You could strip the covers and other extraneous content to allow stuffing more ebooks onto your Kindle. One local Kindle user has automated the process for doing this to his .mobi files. As several people have commented, quite a few covers are larger than the rest of the ebook's content -- my current record holder is an ebook that clocked in at 5.7MB. Sadly, 5.3MB of that was a cover image in a 1800x2700 .png file.

[DNSB]

Originally Posted by eschwartz

And I will say again: the Kindle does not process them slowly, even when you add several hundred books at a time. I have never tested thousands, but I expect it to be no different.

The reason is again as I have already said: the Kindle processes one/a few books, adds them, and repeats. It might also help that I believe metadata is easier to extract from the mobipocket database wrapper than by unpacking an EPUB to get at the OPF. Either way, why does the Kobo slow down the more books you queue and the Kindle doesn't???

What slowdown? Assuming it takes an average of 10 seconds to process an epub added to a Kobo, adding 8500 books is going to take 85,000 seconds or 23 hours. Adding 1 book is going to take 10 seconds.

Even if adding a book dropped to 5 seconds, you are still looking at ~11.8 hours for the 8500 books.

Perhaps you could give us the time required to add 1000 books to your Kindle? Inquiring minds are eager to know.

Also note that Kobo epubs synced from their server seems to take quite a bit less time to process compared to sideloaded epubs though I have only done ~50 ebooks going that route after a factory reset. I don't do that many factory resets so have never put a stopwatch on the process.

[eschwartz]

Originally Posted by davidfor

The metadata that is extracted is:

- Title
- Author
- Synopsis
- Language
- File size
- Format
- ISBN (actually takes the first identifier)

As well as that, the library can show:

- Publisher
- Published date
- Series name and number

Kobo populate these for purchased books from their server, but they have to be populated by other methods for sideloaded books. And I have no idea why Kobo do not extract the publisher and date as it is standard metadata. The cover is not extracted and is generated the first time it is needed.

I have no idea how the above compares to what is on the Kindle.

Kindles just show title/author/cover. TBH I am not sure what I would want to look at the filesize/format/language/ISBN for, sounds like the Kobo is spending too much time extracting and indexing unusual metadata.

filesize is accessible via file browser hacks, if I cared, which I don't, because filesize is not a criteria that affects my reading.
format -- it is a Kindle book. AZW3 can be pretty bad quality and MOBI can be good, so it doesn't even tell me what quality the book is.
language -- I read only English books, but if I read other languages the title would be a giveaway. The Kindle only uses language for choosing a dictionary, and that is based on XHTML lang attributes anyway, just like all the other book content.
ISBN would be a lot more useful if there was any reason to know one outside of cataloging time.

Synopsis -- Amazon expects us to look in the Kindle store for that. I might use it if it was offered, but I am not all that fussed really, it is not *my* priority. And many people across various devices/apps have been using calibre metadata jackets for that.

Series I do rather want. But embedding it in the title is a fair substitute.

Covers are extracted from the book after it is added, I think. I usually sideload via calibre which creates the cover thumbnails anyway.

[eschwartz]

Originally Posted by DNSB

Odd. Around here, I find as many Kindle as Kobo owners who have the desire to have massive libraries on their devices. The only real difference is that the Kobo owners, for the most part, have the option of easily added external storage so it's easier to have massive numbers of books.

That and the squillions of Kindle owners who own three books.

i.e. you said the words "around here". Highly unwise, since MobileRead is in no way representative of Amazon or Kobo's userbase or concerns.

[eschwartz]

Originally Posted by DNSB

What slowdown? Assuming it takes an average of 10 seconds to process an epub added to a Kobo, adding 8500 books is going to take 85,000 seconds or 23 hours. Adding 1 book is going to take 10 seconds.

Even if adding a book dropped to 5 seconds, you are still looking at ~11.8 hours for the 8500 books.

Perhaps you could give us the time required to add 1000 books to your Kindle? Inquiring minds are eager to know.

Also note that Kobo epubs synced from their server seems to take quite a bit less time to process compared to sideloaded epubs though I have only done ~50 ebooks going that route after a factory reset. I don't do that many factory resets so have never put a stopwatch on the process.

Again, it has been some time since I added a ton of books at once. I think I maxed out on a couple hundred.

Adding one book does not take ten seconds, it takes one or two seconds.

Syncing, I can't judge -- the majority of the time has to do with connection speeds.

[davidfor]

Originally Posted by eschwartz

Kindles just show title/author/cover. TBH I am not sure what I would want to look at the filesize/format/language/ISBN for, sounds like the Kobo is spending too much time extracting and indexing unusual metadata.

The Kobo is opening the book to get the title and author. As the rest of the metadata is either in the OPF or comes from the file system, it isn't going to add much time to it.

filesize is accessible via ,,,,,,file browser hacks, if I cared, which I don't, because filesisze is not a criteria tha affects my reading.

Word count, or even page count, would be better as I sometimes choose a book based on size (due to mood, opportunity to read etc). But, I have a fair idea of the book length based on the file size using my experience with the books and the known sources.

format -- it is a Kindle book. AZW3 can be pretty bad quality and MOBI can be good, so it doesn't even tell me what quality the book is.

A format of epub/kepub, PDF, CBR/CBZ, text etc does tell me something - fiction, text book or other info, comic book and "why do I have plain text on the device?"

language -- I read only English books, but if I read other languages the title would be a giveaway. The Kindle only uses language for choosing a dictionary, and that is based on XHTML lang attributes anyway, just like all the other book content.

Same for the Kobo.

ISBN would be a lot more useful if there was any reason to know one outside of cataloging time.

Synopsis -- Amazon expects us to look in the Kindle store for that. I might use it if it was offered, but I am not all that fussed really, it is not *my* priority. And many people across various devices/apps have been using calibre metadata jackets for that.

As the sideloaded books are not directly linked to a shop somewhere, using that method is useless. Plus about half of my read is done somewhere without connectivity, so it wouldn't help. And if I want to look at the jacket, that means I have to open the book which marks it on the Kobo as open. I can unmark it, but, that's extra work. And from what I can see, the reason that calibre metadata jackets exist is because so few of the ereaders support displaying this information separately. There is no way I would buy one that didn't.

Series I do rather want. But embedding it in the title is a fair substitute.

I'll beg to disagree. Embedding the series info in the title is a hack done because the device can't display it separately.

Covers are extracted from the book after it is added, I think. I usually sideload via calibre which creates the cover thumbnails anyway.

The Kobo's generate the cover images as needed. It does waste some space by having different sizes for different places, but it works fine. Or calibre can generate them during the send.

The simple thing here is that the point of the metadata being in the book is that applications can use it. And hopefully in a way that is useful to the user. I am interested in seeing most of the metadata above. I do look at it when choosing a book. So, the time taken in building this is useful to me. If Kobo speed it up, then great. If they add a bit more info, just as good (I'd love to see tags and filter based on them).

[Terisa de morgan]

@eschwartz, sorry, yes, that's the info I was talking about. Not interesting for you, the main reason I won't have a Kindle unless they change library management or I have no other option. Different people, different choices, but it keeps on being more metadata extraction in Kobo than Kindle.

[chrisridd]

That's a good summary @davidfor.

Is there any way to investigate deeper to profile what's happening (at the broad level as we don't have source code) when the Kobo says it is "Processing"? Maybe if someone could telnet to the ereader and run strace?

[shalym]

For me, the biggest difference between the Kobo and the Kindle in terms of processing time is that the Kobo is unusable while new books are being processed, and the Kindle is able to be used.

In other words, I really don't care how long the Kindle takes, as I can still read a book while it is processing the new content.

Shari