[GUI Plugin] Kindle Collections (old)

[meme]

Originally Posted by allisondbl

So. Is there way to remove the dozens of collections that I assume constitute tags you for some reason chose to assign MY books - I certainly did not - without A having to now bloody do THIS one by one by one and B not remove the actual books from the device.

Sorry to hear you are having a frustrating time. Let's see if we can improve things a bit.

The first thing I'd like you to do is make a backup of your collections file on the Kindle so that at least you can get back to where you are now. To do this go to your Kindle's directory, then to the 'system' subdirectory. There is a file called 'collections.json'. Copy that to your PC somewhere.


If you ran the plugin with the menu option "Edit Collections" then the plugin would not have created any collections automatically. It only shows you what is on your Kindle and allows you to manually add books to collections, add/remove collection names.


I suspect you ran the 'Create Collections' option which does create additional collections based on what you have in the Customized settings. The default is to create collections based on your tags that you already have in Calibre (which may have been in the books you loaded into Calibre). These may be the extra collections you are seeing. I think I'll change the default to not create anything in case someone runs Create by accident and aren't aware that books often come with lots of their own tags.

To fix your issue try one of these:

Option 1: Use the menu item 'Edit Collections' to manually select the collections from the drop down box and delete them (using the '-' button next to the collection). This will not delete or modify any books. It only changes the collections and is the best way to be sure you keep/remove the right collections.

Option 2: Use the menu item Customization and select 'Delete' as the action for the tags column. Then run Create Collections again - this should remove the collections that Calibre created. However - if you have manually created a collection that matches a tag name, it would be deleted as well. So only do this if you know your collection names don't match the tag names. If necessary you can copy the collections.json file back to the Kindle to get back to where you were. (Edit: If you use this option, make sure you change 'Delete' back to a blank entry after you run it as kiwidude suggests to avoid creating them by mistake later.)

Option 3: If you look in the 'system' directory on the Kindle you might see a file called 'collections.json.backup.original'. This is the collections.json file that was on your Kindle the first time you ran the plugin - so it should be what you had originally. Copy this file to your PC as a backup. Then copy or rename it to become 'collections.json' in the 'system' directory. Then restart your Kindle. See if the collections are what you had originally.

Remember to restart your Kindle to see the collections on the Kindle if you update them using the plugin.

[meme]

Originally Posted by kiwidude

I believe your particular issue may have been aided by a default I disagree with that meme has in the plugin. That default is to create collections based on Tags. @meme - if you want to put in a default, then by author or author sort is a much better choice imho. What makes it worse, is that the default size of the dialog can mean that the Tags choice is likely off screen until you scroll down.

Yeah, I think I'll remove the defaults from the plugin. With the addition of Edit, it could be confusing whether to run Create or Edit the first time. And with all the tags that get loaded when you import a book into Calibre I can see it could be an issue. There are plenty of details and examples in the first post, so shouldn't try to be too helpful with defaults

[kiwidude]

@meme. Sounds good. The one plus I would say of having "a" default in the list particularly for new users is that it shows the user what they need to populate to set one up without having to read the instructions. There are a lot of columns on that dialog, and obviously if you miss one that is kind of critical you could make a bit of a mess (assuming people are not disciplined to preview before creating collections). I just think that "Tags" isn't the best example, both from being off-screen and for the noise it can create. Author sort is my one of choice (and only). This has the advantage that when you have co-authors for final books in a series, just strip off the co-author in the author sort and you get just one collection covering the whole series, but can still see the display author as being co-written.

[meme]

Originally Posted by nynaevelan

Just to add to this conversation, the only collection which shows up in the collections column in the device view on my k2, is the collection which I created on the kindle. I used to be able to see them also, but I do not know when they stopped showing up as I rarely look at that info but I do remember it was never 100%.

A post by Kovid in another plugin thread may have just inadvertently identified the cause of this issue. Thanks to your collections file the link was obvious.

It seems that Calibre writes the Calibre UUID of a Mobi book into the same field that Amazon stores its ASIN number (to allow syncing to Amazon). When my plugin reads mobi files it uses the code found in that field as the code to put in the collection list. If no code is found it creates a code based on the pathname. Since there is a code in the field, the plugin uses it - not knowing its not an Amazon ASIN identifier - it should really ignore it and just generate the code from the path name.

The Kindle appears to know that the code is not really an ASIN number, ignores it and calculates a new code based on the pathname. So it shows books in collections ok. Since Calibre only knows how to generate the code from the pathname it is looking for that pathname code and not the UUID code.

I just need to modify the plugin to skip the code if its not an ASIN code. The question is whether Amazon ASIN numbers are a fixed format/length. If so I can ignore anything not matching that format. Or if Calibre UUID is a fixed format I can include everything but the UUID type codes. Its not clear if anyone else uses the same asin field besides Amazon and Calibre. I wonder if other Mobi book creating tools populate it.

[kovidgoyal]

The UUID is always a UUID of type 4. i.e. it has a fixed format.

An ASIN is basically a 10 char ISBN, but with the restirctions to make it a valid ISBN removed.

I'd suggest if it is a 10 char strin to assume it's an ASIN otherwise not.

Though I dont recall if Amazon also prefixes the data in the ASIN fields with BOOK or something like that

[meme]

They prepend a '#' at the start of the ASIN, but then so does Calibre. It appears to be '#' followed by 10 characters followed by '^' followed by the book type. The 10 characters seem to start with B - but I'll just check if there are 10 and assume its ASIN as suggested. Thanks.

Now the tricky part is converting everyone's collections from the wrong code to the right code. Not too hard to do, but yet another check for legacy issues

[meme]

It looks like the addition of the UUID to the Mobi file's asin field was introduced in 0.7.44 - which explains why people suddenly started to not see the collections in the Device View after using the plugin for a while...

[meme]

I've investigated this a bit more and wrote the code to change the UUID asin number into a standard pathname hash number. But then I found that periodicals would no longer go into collections!

It looks like I don't need to update the plugin's code after all, but the Calibre code that checks for collections on the Kindle needs to be updated to read Mobi header information.

As far as I can determine, the approach seems to be the following.

The Kindle reads all the files on the Kindle and generates a Collection Code for each of them. If they are of type Mobi, then the Kindle takes the book's ASIN (Amazon ID) value (whatever it is) in the Mobi header, prepends a '#' and appends ^TYPE, where TYPE is the cdetype found in the file (e.g. EBOK, PDOC, NWPR, MAGZ, FEED, None). If there is no ASIN in the Mobi header or the file is not of type Mobi, it will use the SHA1 hashcode of the file's pathname for the Collection Code with a '*' at the front, but nothing added to the end.

When the Kindle puts a book into a collection, it just stores the book's Collection Code. If the Collection Code starts with '#' and ends in ^None (and possibly if its not one of the 5 types), it won't let the book be put into a collection, and won't display it in the collection. The Kindle only checks the ^TYPE at the end of the Collection Code when doing this and ignores the cdetype in the actual file. If the Collection Code starts with a '*' it allows it to be put into a collection.

When the Kindle lists the contents of a Collection, it reads the Collection Codes in the specific collection and then looks at the Collection Codes for all the books to see if there is a match. If there is, and the code does not start with '#' and end with 'None', it displays it.

So basically the asin value in the Mobi is used for collections if it exists, otherwise its the pathname hashcode. Calibre's Device View only knows how to interpret pathname hashcodes and does not look at the Mobi header to get asin/cdetype to create a Mobi format Collection Code.

With the update to Calibre to insert a UUID into the ASIN field, the Kindle automatically uses that ASIN number as the Collection Code to go into collections. If I changed the code to be the hashcode, then the ^TYPE and format would be lost - but worse, since the Kindle checks the ASIN code first when looking for a book to match, it won't find a match.


Hmmm, that's a bit rambling, but it isn't far off how I figured it out The summary is, the plugin will leave collection codes alone, periodicals will be able to be added/managed as before (Edit, well they'll work as they should since I remembered they aren't working yet in 1.4.3..), and Calibre will need to be updated in order to see collections in the Device view. Once I wrap up the new version I'll look to see if there's any support for doing that.

[kovidgoyal]

@meme: Feel free to submit a patch to kindle.driver to get it to read ASINs. I never bothered with it, as using the MOBI reader class for it is slow and I didn't feel like writing special code to extract only the asin.

[zhw007]

Good, thank you

[jay2222]

@Meme => Re/jay2222 - a long time ago you asked about using tags like tgSKI in your titles for items send from Read It Later. Well, with the addition of the title to the customization dialog and the ability to split on a character in the next version of the plugin, you'll be able to create collections for your tags.
==========================

Is this feature available now?

Thanks a million for having kept this request in mind!

[scarlet slipper]

First let me say thank you, meme! Your plug-in helps a lot. I used your screen shots and instructions and created my collections, which worked perfectly. I assumed (wrongly, I think) that whenever I plugged my Kindle into my pc to transfer books, the collections would automatically update. If they should, mine aren't.

I went back to the beginning and if I reread correctly, I need to recreate my collections every time I move books to my Kindle. Can you confirm for me, please, that this is correct? If there's something else I should do, or set up, so that collections updating occurs during transfer, I'd appreciate knowing that. Thanks, again!

[nynaevelan]

Originally Posted by scarlet slipper

First let me say thank you, meme! Your plug-in helps a lot. I used your screen shots and instructions and created my collections, which worked perfectly. I assumed (wrongly, I think) that whenever I plugged my Kindle into my pc to transfer books, the collections would automatically update. If they should, mine aren't.

I went back to the beginning and if I reread correctly, I need to recreate my collections every time I move books to my Kindle. Can you confirm for me, please, that this is correct? If there's something else I should do, or set up, so that collections updating occurs during transfer, I'd appreciate knowing that. Thanks, again!

Scarlet:

It is not done automatically, you will need to decide how often you want to update your collections. Personally, I update my collections after moving books on and off of the kindle, usually about once a week.

Nyn

[scarlet slipper]

Nynaevelan,

Thanks for responding. When you say "update" do you use the create collections option, which I think rebuilds from scratch, or the edit collections option? I'm not quite sure I understand how to edit directly on the kindle.

[nynaevelan]

Originally Posted by scarlet slipper

Nynaevelan,

Thanks for responding. When you say "update" do you use the create collections option, which I think rebuilds from scratch, or the edit collections option? I'm not quite sure I understand how to edit directly on the kindle.

Scarlet:

I only use the create collection option to have them rebuilt from scratch. I've never used the edit feature because I create my collections based on tags so I let the plugin do the work for me.