Kindle device problem with both library and collection

[meme]

Originally Posted by misling

Here is the whole sequence.
Create an empty library...

In the library page, and also on the device page, Half the books, the azw/azw1 show "Main" under "on device" and half, the mobi, do not.

I hadn't noticed it the last time, but this time in the library page a few of the mobi were recognised - about 10 of them.

Also, on the library page the azw/azw1 are in a collection and the mobi are not. On the device page, the mobi are in a collection and the azw/azw1 are not. This is just weird.

All the books are on the Kindle device - that's where I built the library FROM.

This doesn't have anything to do with collections - its entirely an issue of using calibre to manage libraries and devices.

Here's my take on things, but someone else will know much more about how this is actually working,

From a brief bit of testing, I think your metadata.calibre file is out of sync. You should eject and disconnect your Kindle, shutdown calibre, re-connect the Kindle then rename the metadata.calibre file on the Kindle to something else (e.g. metadata.calibre.old), then restart calibre. This will force calibre to recreate the metadata.calibre cache file and will take several minutes. Once it is done you should see your books correctly marked as on-device/in-library.

When I created an empty library and imported my books I found some not marked as in-library and others ok. Even for books that weren't duplicates. However, I found that if I did a Send to Device to resend the book from calibre to the Kindle, the on-device/in-library status was ok.

Once I removed the metadata.calibre cache file and had calibre recreate it, all my books were correctly marked in the new library.

However, when I switched back to my old library, some books that were on the device were no longer marked as on-device.

I suspect the issue is related to using 2 libraries and one device. Or more specifically in the way books are added to multiple libraries. Each book is given a unique calibre id that gets compared to what is in metadata.calibre and it may be the id isn't always the same between libraries in some cases.

I'm not sure if creating an empty library and importing your Kindle books is the best approach to cleaning things up - its not clear if you lose any calibre specific data in that process, but I'll leave it to others for suggestions in that area.

[misling]

Originally Posted by meme

This doesn't have anything to do with collections - its entirely an issue of using calibre to manage libraries and devices.

Here's my take on things, but someone else will know much more about how this is actually working,

From a brief bit of testing, I think your metadata.calibre file is out of sync. You should eject and disconnect your Kindle, shutdown calibre, re-connect the Kindle then rename the metadata.calibre file on the Kindle to something else (e.g. metadata.calibre.old), then restart calibre. This will force calibre to recreate the metadata.calibre cache file and will take several minutes. Once it is done you should see your books correctly marked as on-device/in-library.

When I created an empty library and imported my books I found some not marked as in-library and others ok. Even for books that weren't duplicates. However, I found that if I did a Send to Device to resend the book from calibre to the Kindle, the on-device/in-library status was ok.

Once I removed the metadata.calibre cache file and had calibre recreate it, all my books were correctly marked in the new library.

However, when I switched back to my old library, some books that were on the device were no longer marked as on-device.

I suspect the issue is related to using 2 libraries and one device. Or more specifically in the way books are added to multiple libraries. Each book is given a unique calibre id that gets compared to what is in metadata.calibre and it may be the id isn't always the same between libraries in some cases.

I'm not sure if creating an empty library and importing your Kindle books is the best approach to cleaning things up - its not clear if you lose any calibre specific data in that process, but I'll leave it to others for suggestions in that area.

So your idea is that if I use my main library for this, and not create a new library, and recreate the calibre metadata on the device so that it is all from one library, it would solve this problem... maybe so, that does make sense... Too bad, because I wanted JUST the stuff on the Kindle to work with, which is why I had created a new library. The main library is so huge that it's hard to work with. Still, whatever works
Hadn't known that each library gives a different ID to a book when it pulls it in. That makes moving things around between libraries less useful - maybe it's why I get duplicates every so often when transferring books between my 'work' library and my 'finished' (main) library, even when the source book is the same. I hate to turn on the automerge, because then different editions merge, and I don't want that. So I merge by hand. But that's a different issue. Sorry, getting sidetracked.

I'll try to attach the screenshots in case anybody wants to see them. Next post. This one is getting long.

[misling]

Screen shots:

I'll try that idea of using just the main library and recreating the calibre metadata when I get home from work this evening. Thanks for the suggestions. Oh and THANKS for the advice how to add screenshots. Hadn't noticed that advanced button.

[misling]

OK , I tried deleting the calibre database off the kindle via usb and then booting up calibre and seeing what it would find in the main library.

The kindle has 1654 objects on it when I list on the kindle by title, of which 38 are active content (games & stuff) and I know those won't go in the calibre list. That leaves 1616 non-active things on the kindle. There are also 62 collections names, 32 news reader feeds (5 magazines, several different dates) from magazine web pages, added through calibre news download, and 1 magazine subscription digest - I don't know if those are in the count?

In Calibre I get that 1528 items are in the 'main' 'on device' column of the main library. I can't tell what's missing if anything. But the numbers might add up if calibre ignores some stuff like active content and collection names and news feeds. Since there are over 8000 items in the calibre main library it's hard to tell if anything is missing from the list.

Would it work to delete the calibre database thing from the kindle and then create the new empty library and then add to the empty library from the device? This is what I was doing before, only I didn't delete the calibre database from the kindle first. Would this make the empty library accurate?

It's hard to test out this stuff while working around my day job and evening commitments. Work's been demanding this week, and I can barely stand to boot up the computer in the evening after I get home from working on the computer all day (I develop & support hospital software). But I'd really like to make it work... doing collections by hand is VERY time consuming.

I don't mean to be bitching about Calibre. It's a great piece of software. I love it.

[misling]

Originally Posted by misling

Would it work to delete the calibre database thing from the kindle and then create the new empty library and then add to the empty library from the device? This is what I was doing before, only I didn't delete the calibre database from the kindle first. Would this make the empty library accurate?

In case anybody was wondering, this does work.
I suppose not very many people use multiple calibre libraries, and that's why this isn't a problem for most people. To properly fix this without having to delete the calibre database from the kindle every time you change libraries, I think you would need a separate calibre database on the kindle for each library, instead of one calibre database on the kindle.