dupe books problem

[AlanHK]

I make AZW files on my PC and sideload to my Kindle PW3. Never download wirelessly or from Amazon.

I use Kindlegen in Sigil; it needs a pseudo ASIN when it makes the AZW. I create set one using an abbreviated form of the title.

All good the last 3 years. But I had two different books with the same title, and used the same code. Both were showing with the same cover, and who knows what else was scrambled. I deleted them on the kindle, then deleted the associated files in the Documents and thumbnail folders on the Kindle. Thought I was purging these books; obviously I screwed up there.

Rebooted it a few times. But now when I sideload new books, they do not appear, and are not findable by search. And one of the books I deleted does show in the library with a generic cover, but when I select it, it's "Item Not Available: has been removed and no longer available on you kindle", when I delete it from the screen, it's still listed.

I presume I've screwed up the index. (Where is it?) If that's it, should/how can I force it to reindex? Would be annoying, I have about 1000 books but I'll do it if necessary.
Or any other option?



Absolutely last option is reset and reload all my books. I have dozens bookmarked and with notes.

[jhowell]

Have you tried restarting your Kindle?

(Edit: Never mind, I missed that in the original post.)

[j.p.s]

kindles keep track of books in an sqlite database in /var/local/cc.db which is only accessible on jailbroken kindles. Indexing only has to do with locations of words in books.

Doing a reset (as opposed to restarting (rebooting)) should clear your problem, but it will also reset all settings to default and might lose indexing.

[AlanHK]

Originally Posted by j.p.s

Doing a reset (as opposed to restarting (rebooting)) should clear your problem, but it will also reset all settings to default and might lose indexing.

What do you mean "might lose indexing"? Surely all book files are deleted and the index zeroed?

Is there any alternative to "factory reset"?
With that I have to re-register with Amazon, load and organise 1000 books... after writing down the bookmarks and notes in the books I want to keep. So I'm not looking forward to that.

(These are all sideloaded books, aside from the dictionaries, there is no cloud backup or whatever Amazon does.)

It's still working apparently normally, except the two ghost books and inability to load any new ones.

[j.p.s]

It has been a long time since I have done a reset and do not recall whether books are erased by a reset. I would have guessed they are not. Someone with direct experience will have to chime in. In any case, you can preserve all your bookmarks and notes by making a tar or recursive zip (-r option) of the contents of the documents directory on the kindle.

As for an alternative, as I wrote before, jailbreak. Preople always think they would never have any use for jailbreak but sometimes it turns out there is not only a use but a need.

Back to the reset option. If you have no books from amazon, why do you need to register? in any case, reregistering should be quick and easy and there is no need to reorganize your books or manually redo your bookmarks and notes (if you copy all the contents of the documents directory).

[AlanHK]

Originally Posted by j.p.s

It has been a long time since I have done a reset and do not recall whether books are erased by a reset. I would have guessed they are not.

The only reset I can see is the "factory reset" and pretty sure that will wipe out all books.

Originally Posted by j.p.s

Someone with direct experience will have to chime in. In any case, you can preserve all your bookmarks and notes by making a tar or recursive zip (-r option) of the contents of the documents directory on the kindle.

I was wondering about that, as I assume all the status info for each book is in its sdr folder.
So if I back up the Documents folder, then reset the Kindle, then copy the the whole Documents folder back, it will restore all the bookmarks, notes, page position, collections, while rebuilding its central database of titles so I can load new books? Worth a try, if I've reset anyway.

Originally Posted by j.p.s

As for an alternative, as I wrote before, jailbreak. Preople always think they would never have any use for jailbreak but sometimes it turns out there is not only a use but a need.

What does jailbreak let me do to solve this?

I'm on 5.9.2, and there was no jailbreak for that when I got it, though I think there is now. I put in the block folder to stop automatic updates to give that a chance to evolve. So I was thinking I would do that if I have to rebuild everything anyway.

Originally Posted by j.p.s

Back to the reset option. If you have no books from amazon, why do you need to register? in any case, reregistering should be quick and easy and there is no need to reorganize your books or manually redo your bookmarks and notes (if you copy all the contents of the documents directory).

I think I had to do that to turn off ads. And to download dictionaries, though I can presumably back up and restore the ones I have. And I may want to actually get am Amazon book, at some time. But not too worried about that.

[j.p.s]

Originally Posted by AlanHK

The only reset I can see is the "factory reset" and pretty sure that will wipe out all books.

OK. I think you are correct about that.

I was wondering about that, as I assume all the status info for each book is in its sdr folder.
So if I back up the Documents folder, then reset the Kindle, then copy the the whole Documents folder back, it will restore all the bookmarks, notes, page position, collections, while rebuilding its central database of titles so I can load new books? Worth a try, if I've reset anyway.

I gave up on collections years ago and know very little about them, but that might be the fly in the ointment. You say amazon has none of your content. If that means they also don't have a record of your collections, then your collections will disapear with the reset. I don't think any collection information is in the .sdr directories.

You might want to start a new thread with collections in the title then ask about getting collections back after a reset, explaining that you did not use email, send to kindle, etc. and which firmware you have. (amazon has changed collections a lot over the years.) There are quite a few people that know quite a bit about collections.

What does jailbreak let me do to solve this?

I'm on 5.9.2, and there was no jailbreak for that when I got it, though I think there is now. I put in the block folder to stop automatic updates to give that a chance to evolve. So I was thinking I would do that if I have to rebuild everything anyway.

You have a choice of 2 jailbreaks, the 5.9.6.1 specfic to the PW3 and the most recent all kindle jailbreak that requires inclusively between 5.10.3 and 5.13.3, the former requires downloading and installing the factory firmware will update to 5.9.6.1 and the latter (known as KindleBreak) is much faster and simpler to do and does not require using factory firmware as an intermediate step.

Once jailbroken, you can use a sledgehammer or a surgical approach. The former is to remove /var/local/cc.db and let the kindle rebuild it, but I think you might lose collections doing that.

If you are willing to manipulate an sqlite database, you can likely repair whatever messed up the rows for the problematic books.

If you would like to try this approach and can not figure out what to do by inspecting the entries for your books, there are quite a few people in the Kindle Developer's Forum with the expertise to help.

You can use SQL commands directly or one of the sqlite browser apps available for just about any platform. (You'll need to copy /var/local/cc.db to your computer.

amazon added some custom SQL function to cc.db so accomodating that will involve some tedium.

I think I had to do that to turn off ads. And to download dictionaries, though I can presumably back up and restore the ones I have. And I may want to actually get am Amazon book, at some time. But not too worried about that.

OK. Well, it is easy enough to register and I don't think either of the above jailbreak methods loses your registration.

[AlanHK]

Originally Posted by j.p.s

I gave up on collections years ago and know very little about them, but that might be the fly in the ointment.

Not a big deal if I have to re-sort the books manually to collections.
I'll do the one for currently reading and sort the rest to genres later.

Originally Posted by j.p.s

Once jailbroken, you can use a sledgehammer or a surgical approach. The former is to remove /var/local/cc.db and let the kindle rebuild it, but I think you might lose collections doing that.

If you are willing to manipulate an sqlite database, you can likely repair whatever messed up the rows for the problematic books.

I did a little SQL in the 70s, but even if I could get up to speed, I think less work to just let it rebuild from scratch, if the only info lost is collections.

[j.p.s]

Originally Posted by AlanHK

Not a big deal if I have to re-sort the books manually to collections.
I'll do the one for currently reading and sort the rest to genres later.

OK. Then the quickest and easiest fix is to:
backup documents directory
"factory" reset
restore documents directory

But since you manupulate metadata in ways that risk fouling cc.db I think it would be worth your while exploring the contents of cc.db to get insight into what is going on.

If you jailbreak, you can rename cc.db and let your kindle rebuild it. If something goes wrong, you can rename the original back to cc.db

You could also make a backup copy of cc.db before doing something risky. If you retrigger the problem, you could remove the offending book then restore the backup cc.db

I will also reply with more information in your Developer's Forum thread, but it will require some research and I have some pressing requirements on my time.

I did a little SQL in the 70s, but even if I could get up to speed, I think less work to just let it rebuild from scratch, if the only info lost is collections.

I think your previous SQL experience will make it easier for others to help you track down the root of the problem.

[Little.Egret]

I recall that doing a unregister/ register on a Kindle is halfway between restart and factory reset. No content is lost, certainly not sideloaded.

[AlanHK]

Originally Posted by Little.Egret

I recall that doing a unregister/ register on a Kindle is halfway between restart and factory reset. No content is lost, certainly not sideloaded.

Yeah, but I don't think that will rebuild the database, which is my problem.

So I'll see if I can jailbreak it and then delete the database.

[AlanHK]

Originally Posted by j.p.s

I think your previous SQL experience will make it easier for others to help you track down the root of the problem.

I know what root of the problem is, I had non-unique ASINs, which presumably are keys in the database. Tried to fix it by deleting the dupe books. That just made things worse...

[j.p.s]

Originally Posted by AlanHK

I know what root of the problem is, I had non-unique ASINs, which presumably are keys in the database. Tried to fix it by deleting the dupe books. That just made things worse...

Deleting a book does not delete it from the database.

My guess is that deleting the two rows with duplicate identifiers from the database will take care of the problem, but there is no way to know without having a look. One of NiLUJe's packages has a kindle sqlite that can manipulate the database, but any sqlite can list it.