Tips for loading a large library onto a KO2

[Strether]

I found that by restarting the Kindle and plugging it into the wall before I went to bed, that it had indexed 750 books by morning. FWIW.

Jim

[HarryT]

Ok, I’ve finally, after nine days, finished loading my library (or what I want to have on my Kindle, at least) onto my KO2. A total of 8636 books occupying 20GB of storage. That’s an average of 2.5MB per book, which is high because the books include a hundred or so VERY large Delphi omnibus editions. The largest book I have loaded is a little over 200MB in size. My Kindle still has 7GB free storage space.

So, how does it work with that many books on? Generally speaking, pretty well. Paging through the library is fast (all 1080 pages of it!), and searching for book titles returns all matching titles pretty well instantly. No problems reading. No crashes, hang-ups, or any other strange behaviour.

Some issues, which are a consequence of library size:

One issue is that searching in books takes quite a long time. Not too surprising with 20GB of content to search. Eg if I open Joe Abercrombie’s “First Law Trilogy” (a very large book!) and search for the word “sword”, it takes 64s to return the 546 matches. Edit: the search speed is constant; searching for any word or phrase in any book, long or short, takes 60-65s. Must be related to the total size of the index, I guess.

Doing a global search for words across all books just plain doesn’t work. It probably would return results eventually, but it hadn’t done so after waiting 10m. Again not too surprising, and this is a facility I don’t use.

As I mentioned above, searching for book titles or authors works fine and returns results pretty much instantly. Eg if I search for the word “hope”, it finds the 28 books in my library which contain “hope” either in the author’s name or the title in less than 2s.

Another issue is that when you connect to the PC, and add or delete books, when you disconnect the USB cable, the library takes about 10 minutes to fully update. Just a question of searching through that many books to see what’s changed, I guess.

So, an interesting experiment. You can put 8500 books on your 32GB Kindle (you could put 3x that number of normal-sized novels, easily) and it works fine as a reader, but word search within a book is quite slow, and global word search effectively unusable. Realistically, you can load about 1000 books a day, allowing for indexing, so take a month off if you plan to load your 30,000 books!

Is it worth doing? On balance, yes, for me personally. Global word search is something I don’t use, so the lack of that is no big deal for me. I’m happy with the outcome - I can finally have as much of my library as I want on my Kindle.

Hope this experiment is of some interest to others! Now to actually do some reading . I’ll keep track of battery life, and see how that goes.

[93terp]

Thanks for this info Harry. It’s very helpful.

[Readirect]

It's worthwhile noting that the Kobo OS indexes books virtually instantly. You'd think that Amazon might want to hire a Kobo programmer to learn how to do that.

[HarryT]

Originally Posted by Readirect

It's worthwhile noting that the Kobo OS indexes books virtually instantly. You'd think that Amazon might want to hire a Kobo programmer to learn how to do that.

It's doing a different thing, though, is it not? Last time I owned a Kobo, it did not have any sort of global search facility.

[disconnected]

Originally Posted by Readirect

It's worthwhile noting that the Kobo OS indexes books virtually instantly. You'd think that Amazon might want to hire a Kobo programmer to learn how to do that.

It isn't the same thing though, is it? If I do a search from the home page on my Oasis I can have it list every book the search term appears in and then show me every occurrence of the term inside any of the listed books. I don't think my H2O2 can do that. It isn't a feature I use much but I can see that someone might find it useful.

Never mind. Harry types muvh faster than I do.
And more accurately.

[DNSB]

Originally Posted by HarryT

It's doing a different thing, though, is it not? Last time I owned a Kobo, it did not have any sort of global search facility.

On my KA1, search inside a book or by title/author/series in the library is virtually instantaneous. No global search for a word in all books though it would appear that Amazon's implementation is borked by having too many books indexed. Hmmm... I wonder what having books with every word lovingly indexed is doing to the database size with large collections.

[Sirtel]

Originally Posted by DNSB

On my KA1, search inside a book or by title/author/series in the library is virtually instantaneous. No global search for a word in all books though it would appear that Amazon's implementation is borked by having too many books indexed. Hmmm... I wonder what having books with every word lovingly indexed is doing to the database size with large collections.

Once the books are indexed, searching is very fast, including global search. True, I don't have several thousand books on any of my Kindles, but I do have well over a thousand.

[John Hopper]

Originally Posted by Sirtel

Once the books are indexed, searching is very fast, including global search. True, I don't have several thousand books on any of my Kindles, but I do have well over a thousand.

I have nearly 2,500 books on my Kindle and global searches are effectively instantaneous.

[HarryT]

Originally Posted by John Hopper

I have nearly 2,500 books on my Kindle and global searches are effectively instantaneous.

Yes, I have a similar number on my Voyage, and find the same. There seems to be some point between 2500 and 8500 at which the poor search engine throws up its hands in horror and gives up .

[montalex]

Thanks for that, Harry! I still read on my old Kindle Keyboards, because they are fine readers and have open 3G so I can get my email when I'm out in the woods, but they are slow at indexing, and if I ever tried to put 2,000 books on one of them, it would burst into flame and melt into a puddle. Instead, I back up my Calibre library to a cheap 8GB usb drive. About 4,000 books for the old age home! It's called hoarding...

[DNSB]

Originally Posted by Sirtel

Once the books are indexed, searching is very fast, including global search. True, I don't have several thousand books on any of my Kindles, but I do have well over a thousand.

Forget this reply. HarryT already replied about global search borking on his library.

[June]

Thank you HarryT for this info, very helpful!
:getting ready to start transferring 2500+ books:

[j.p.s]

Originally Posted by HarryT

I've found that it seems to go a lot quicker if you keep the Kindle awake. It seems to slow to a crawl when the Kindle goes to sleep. When I was doing the batch of books in my previous post, I had the Kindle plugged into my laptop's USB port, but "ejected" (ie it was powered, but not in USB drive mode), and I refreshed the page every 5 minutes to keep it awake.

Maybe using the ~ds search string will disable sleep and allow indexing of the most books possible overnight. I just learned of this string yesterday and do not know whether it merely disables the sleep screen being displayed or actually disables sleep.

Of course, if it disables sleep, the battery will drain faster if it is not being charged. Also, the only way that I know to revert the ~ds state is to restart the Kindle, but that seems a small price for maximizing indexing rate for a large number of books, if it works.

[Cootey]

Originally Posted by Barty

Amazon needs to address the hang while indexing problem. The only reason I would buy the 32gb is to dump my entire library in one go. If I have to dribble it out like this I'm not going to bother.

My habit is to keep a few books I'm reading on the kindle. After a while I accumulate about 100 on the device, but that's about it. Occasionally, in a fit of enthusiasm I may load 5, 10 at a time, and it's a pain when the indexing hangs and I have to delete and load one at a time to see which the offending book is.

The software needs to monitor the process and if it determines the indexing on a book isn't progressing as expected to put it at the end of queue and move on. It should show you the book(s) being indexed and % completed and let you stop indexing on some books and maybe just let them go unindexed.

Here we are, almost a year later, and this one post succinctly states how things should be, yet not a single one of these improvements have been in multiple software upgrades. I know. Amazon doesn't read these forums, but Barty's suggestions are fairly obvious fixes. I've only got 300+ books, not the thousands some of you have. I bought a brand new Paperwhite gen 7, grabbed EVERYTHING, then have been pulling my hair out trying to figure out why the thumbnails haven't loaded after two weeks. I didn't have this problem with my Paperwhite Gen 6. Thanks to Harry's discovery, I'm now s l o w l y downloading 10 books at a time.