HD Issues with Searching/History

[watani]

Lately I've been having problems with my Kindle Fire HD with regards to various search and history features that seem to be related to the Kindle's index and I've been struggling to find a fix.

One issue is that the search index for some of my books seems to be broken. Only some of the instances of a word are listed in the search results, rather than results from the entire book. One specific example is that one long ongoing webnovel (1200+ chapters) only returns results from the final 50 chapters. Searching was fine for the original book I uploaded, but seems to have broken after I uploaded an updated version with more chapters. The book itself shouldn't be corrupted since opening it up in Calibre ran into no problems, the bug check didn't find any problems, and search results are correct in the Calibre internal viewer. I tried forcing the Kindle to re-index the book by deleting it, rebooting the Kindle, then reuploading it, but somehow the Kindle remembered my previous interactions with the book through the reboot/reupload.

The other issue started up a few months ago where now my Kindle resets all book statuses as "New" in the library/carousel every time I upload a new book through Calibre. The Kindle still remembers my progress in each book when I click into each book, but the library/carousel completely loses track of which books I read last prior to the new uploaded books and instead orders all my books by upload time when sorting by "Most Recent".

Anyone have any advice on what I should do? I am willing to do a factory reset as a last resort, but I'd like to know if there's a less drastic fix I should try.

[stumped]

bite the bullet and do the factory reset
amazon will not help you because the issues are with sideloaded books, and the way the fire manages and stores indexing is well hidden

things you could try first, that may help is
1.get to the list of (system) apps via settings, and delete all the cache and data you can find for anything to do with the books' apps
2. there is a soft reset option- less destructive than factory reset. hold down power button and ignore all messages until tablet restarts ( 30 - 40 secs). that will not wipe any books or apps.

with the Fire HD ( which is no longer called Kindle fire BTW) stuff like kindle reader, prime video etc are apps, albeit pre-installed ones that you can''t delete. so you can do "app" stuff like clear cache, data. ( that's a known fix e.g. for issues with prime video)

you other option, if you don't want to factory reset, and you mostly sideload, is to drop the whole Kindle thing. install google services, convert all your books to epub, then use an epub reader of your choice - Moon reader is good. That give you far more control.

you don't say what generation fire HD but anything running Fire OS 5 will allow you to install the 4 apps needed for google play store to work, and instructions are in other threads.

a slight concern is that you mention the carousel. that is a feature of OS4, i.e. the ~2013 devices, and there is no easy way to add google stuff to those.
check amazon / wikipedia & work out exactly what device you have, and look in settings to see what version of Fire OS is running. and chack that you have all available system updates.

to help with troubleshooting, explain exactly how you do the sideload from calibre steps

[watani]

Originally Posted by stumped

bite the bullet and do the factory reset
amazon will not help you because the issues are with sideloaded books, and the way the fire manages and stores indexing is well hidden

things you could try first, that may help is
1.get to the list of (system) apps via settings, and delete all the cache and data you can find for anything to do with the books' apps
2. there is a soft reset option- less destructive than factory reset. hold down power button and ignore all messages until tablet restarts ( 30 - 40 secs). that will not wipe any books or apps.

Thanks for the help! I kinda figured that a factory reset might be the only way to guarantee fix the issue. But regarding your two suggestions:
1. I've looked up the settings and couldn't find a way to clear any cache/data under the Reader Settings
2. I've already tried the soft reset option a few times. Originally I thought it didn't work because one of the books which I deleted prior to the soft reset somehow managed to retain its history through deletion -> soft rest -> reupload. A subsequent soft reset following the deletion of 1/3 of the books on my Fire HD might have worked however, but I'll have to keep using the Fire HD to know for sure.

Originally Posted by stumped

a slight concern is that you mention the carousel. that is a feature of OS4, i.e. the ~2013 devices, and there is no easy way to add google stuff to those.
check amazon / wikipedia & work out exactly what device you have, and look in settings to see what version of Fire OS is running. and chack that you have all available system updates.

to help with troubleshooting, explain exactly how you do the sideload from calibre steps

Sorry, I think I misspoke when I said carousel (I was using the terminology from an older Kindle HD that I previously used that was ~2013). I believe it's more appropriate to call it a list of the 7 most recent books read.

My Fire HD is a Fire HD 8 (7th generation) with Fire OS 5.3.7.0 that is up to date as of this morning.

As for how I sideload from calibre, here are the steps:
1. I take the epub and convert it to azw3 inside calibre
2. I connect my Fire HD to my computer and wait for the books/metadata to be read (usually takes about 2+ minutes)
3. I select books I want to upload and choose the "Send to Device" button in the main toolbar.

Note: I use the following metadata plugboard for my sideloading:
azw3:MTP_DEVICE = {#category:|| > }{series}{series_index:0>2s| #| – }{title:| |}

As an additional sidenote/issue, I've noticed that Calibre has a tendency to abruptly crash with no error when the USB connection to the Fire HD is interrupted even slightly. I also have crashes if I connect the Fire HD to the computer first and then open Calibre. I've heard this issue with certain Win10 systems not working well with Calibre was not reproduced by the Calibre dev, so I'm not expecting an answer to that problem.

[stumped]

that helps
you are not the only one to report issues using the wired interface
i believe calibre puts 2 files onto the device, for storing a record of transfers. Deleting them & having it treat tablet as a new device may help- it will do a long initial scan then recreate them [ I dont have those file names or locations to hand but they probably have the word calibre in them
the other suggestions are also all valid. including adding google services to get more options
probably you have 2 separate issues - a flakey usb connection ( which you could bypass by using the calibre companion wireless app instead), and a separate issue with indexing and history which is not a calibre issue, its is mol likley otehr corrupt resports on the device e.g. in the relevant app data.
try the soft reset and clear app data / caches and see if that helps

FWIW, usual usb troubleshooting is try different cables, different USB ports on PC, or try via a USB hub. Wireless connections may be much less troublesome

[stumped]

look for these 2 files on the fire tablet
metadata.calibre and driveinfo.calibre
if the USB connection is breaking while calibre is doing the transfers, one of both may be corrupt.
delete them and re-test

[watani]

So I've gone ahead and deleted all of the books I've uploaded to the Fire HD 8. Following that, I performed a soft reset and cleared the cache partition. (I haven't found any way to clear the data of the reader app, however.) I then uploaded about 20 books onto the Fire HD 8 through Calibre.

Now when I try to perform searches the Fire HD 8 just shows a spinning ring after I select the word to search. As for the word suggestions, sometimes it shows the list of autocomplete words after a bit and other times it also gets stuck on the spinning ring.

Is this a sign that the indexing is hopelessly corrupt and only a factory reset will solve the issue? Or just that the indexing is ongoing and needs to complete? Some of the books I uploaded are 5000+ pages according to Calibre's Count Pages plugin, so they're pretty long.

One worrying thing is that I'm noticing that my search history is being saved for the various books even though I fully deleted the book and performed a soft reset. That indicates that the Fire HD 8 is still somehow saving information about the books through the soft reset and cache clear.

What's strange to me is that in the past, when I uploaded a book through Calibre and tried searching it would usually immediately start populating results. Or is that a false impression that I got when the book had already been indexed between when I uploaded it and started reading?

Edit: After choosing a word and letting the Fire HD 8 sit searching a book (1600 pages) for 10 minutes, it finally returned a full list of results of that word. Does this mean indexing only occurs while I have the book open?

[stumped]

a don't use the indexing/ searching feature - so other my know more
what i do know
with an e-ink kindle , indexing is slow, and one problem book can stall it , preventing all future books from indexing properly. ( the only fix for that is trial and error to find the problem book). you added 20 large ones, maybe one or more of those is teh culprit.
it would be tedious but productive to add them one at a time and check after each one.
the fire tablet probably uses the same indexing code, but is on a device with a faster processor. it probably looks instant when you sideload a few normal sized books, but one big problem book could cause what you now see. its because kindle has the concept of one huge index across all books, most other reader apps do that on a per book basis. AFAIk, indexing of kindle format books has to happen on the tablet

you have proven that search history survives a soft reset/ cache clear. so only a full factory reset is going to clear that ( unless a de-register, re-register will clear it)

i don't recall ever reading what it is with some books that the indexer cannot cope with - maybe unusual characters or code ?

[watani]

Alright, thanks for your help. I've gotten a better understanding of the Fire HD indexing after some experimenting.

1. It seems like indexing doesn't occur until the book is opened (or maybe "priority indexing" occurs when a book is opened for the first time).

2. A first pass occurs which gets all unique words in the book (which populates the autocomplete list) after which later passes get the instances of the words.

3. Results may be listed even though the indexing isn't completed or has been interrupted (Test: I couldn't find a specific phrase near the end of a nearly 2 million word book but I can find results through about 9/10 of the book).

In any case, it may just be the case that the indexer gets really confused with extremely long books and I shouldn't rely on it in the future (I can just search things up using the Calibre reader on my computer instead). At least the status of books being properly recognized by the Fire HD as previously read might have been fixed over the course of all this.

[stumped]

for the record, in case anyione is still following along. the indexer you are dealing with is something within the Kindle reader app, built in to the tablet ?
and it kicks in when you open an ( amazon format) book, which can be a sideload?

i had done some searching and had read that the e - ink kindle is different - it indexes the entire device whereas the fire indexer builds an index per book - is that the case ?

also , a kindle has hidden index.db file but I could not find anything about the fire tablet having such . a decent file explorer app will let you search for hidden files and folders