[GUI Plugin] Kindle Collections (old)

[koland]

Originally Posted by meme

First - you can definitely see all the books on your Kindle and what collections they are in - and whether or not they are in Calibre - use Edit Collections.

I'll give that a try again - I was afraid to use it, after I managed to end up with only the Calibre maintained collections last night. Of course, I was able to go back a few days and get a good JSON file to restore, then start over. Also, the last time I managed to get into the window (I was getting errors, for a while), I could see the collections created in Calibre, but no books; all I could seem to do was to delete collection names on the Kindle.

Second, I think I'm confused

If you really only have 2 collections

LOL!

No, not only two (or only two pages). But, I ended up with the original collection name (no "_" in front) and the Calibre maintained one (with "_"). I have a field for the imported collections and use it to edit and create new (the really old version I had used had used my tags ... all of them, which created a huge mess; removing those, while not losing the assignment of non-Calibre collection books (a couple thousand, or so), has been a priority.

So, now I have two "SciFi" (and two Fantasy and two Currently_Reading, etc) collections and want to merge them, without copying the Kindle only books into Calibre.

...and you gradually want to move to one collection that is Calibre managed, then you may want to:

1. Import Collections into Calibre into a custom column (e.g. "Kindle Collections") (only books that are in calibre will get the collection name put into the column)
2. Setup the plugin's Customize to Create collections based on your custom column
3. Run Preview (to test/check) then Create - only the names in the custom column will be updated.
4. Now when you import a book into Calibre you can add your collection name into the "Kindle Collections" column and run Create, and manually delete (using Edit Collections if you want) the entry in the Kindle-only collection.

Which is what I have, so far...

When using Create Collections, it will only update the names it finds in the columns you setup in Customize, and leave your Kindle-only collection names alone. To be extra safe, you can put the Kindle-only collection name in the "Never delete" text box on the Customize screen and it will never be modified (and will be ignored when checking for changes which will be helpful if you really do want to mix automatic and manual creation).

So, it sounds like I am stuck with two of each collection, until I manage to copy those books into Calibre, process them to fix titles, metadata, etc, assign the "new" collection in the custom column, then export them back to the Kindle (is this step necessary?) in order to add them to the new collections.

What I cannot do (if I understand correctly): Leave 200+ books in the Kindle collection, add the Calibre maintained books to it, import the collections and then create on the Kindle ... as this would remove those 200+ books from the collection, as they are not in Calibre, so would not be added to the collection when it is created. Correct?

I don't want to mix manual and automatic creation, so much as use automatic creation, have no duplicate collection names (other than the "_") and have all my books in collections (whether downloaded from Amazon or managed by Calibre).

If I can't do that, it's ok. Just means a substantial amount of work before my collections are working again (darn those free books + the bad programming design that slows the Kindle as the number of books installed rises). I've been avoiding putting the DRM'd Kindle only books into Calibre, since they are useless to read there and can't be sent via wifi to the Kindle of my choice (or read on one of my other readers).

[koland]

Originally Posted by bpalesch

My question is NOT how do I create categories. The plugin is doing that correctly. My question is, when I upload a book from Calibre to my K3, does this plugin provide a way to put the book into the correct category on the K3? Or does it ONLY/JUST create categories on the Kindle? What's happening now is that when I upload books and re-run the plugin I get the correct collections, but the books are NOT being put into them; they're being listed after my collections. I have sort by collections selected.

I addition to running the Create after copying books to the K3, you must also, immediately after ejecting the Kindle, do a RESTART (menu, settings, menu, restart). I find it helps to put the Kindle into a "by title" sort first, so it doesn't play with the collections as soon as it is ejected.

[bpalesch]

Originally Posted by koland

I addition to running the Create after copying books to the K3, you must also, immediately after ejecting the Kindle, do a RESTART (menu, settings, menu, restart). I find it helps to put the Kindle into a "by title" sort first, so it doesn't play with the collections as soon as it is ejected.

Doing the restart. Tried with Kindle on Sort by Title. Reran everything - no joy. Guess I'll just have to put up with doing it manually. Would have been nice if I could have gotten it to work though.

[meme]

Originally Posted by bpalesch

Doing the restart. Tried with Kindle on Sort by Title. Reran everything - no joy. Guess I'll just have to put up with doing it manually. Would have been nice if I could have gotten it to work though.

It should definitely work.

When you run Preview - what does the report show?

Do you have data in the "Collections" custom column against books that are both in Calibre and on the Kindle?

[meme]

Originally Posted by koland

I'll give that a try again - I was afraid to use it, after I managed to end up with only the Calibre maintained collections last night. Of course, I was able to go back a few days and get a good JSON file to restore, then start over. Also, the last time I managed to get into the window (I was getting errors, for a while), I could see the collections created in Calibre, but no books; all I could seem to do was to delete collection names on the Kindle.

The no books in collection could have been due to an encoding error in the pathname. I'm wondering if there is still some aspect of this leftover for you and bpalesch. If you go into Edit Collections you would see a ton of Books with No File's listed (there are usually a few) - this is where the code for a book that is in the collection file doesn't match the code the plugin is generating (or the book was deleted). Do you know what text encoding you are running (end of the Preview report should show it).

No, not only two (or only two pages). But, I ended up with the original collection name (no "_" in front) and the Calibre maintained one (with "_").

When you use the Create Collections option the plugin will ignore any prefix when comparing collection names - so if you look in the preview report you might find both names (with and without _) are being considered the same and being replaced by the _ version. There is an option in Customize that you can uncheck to tell it to not ignore the prefix when comparing names. Usually this is left on so you can change prefix/suffix easily - but in this case you might need to set it to ignore.

I have a field for the imported collections and use it to edit and create new (the really old version I had used had used my tags ... all of them, which created a huge mess; removing those, while not losing the assignment of non-Calibre collection books (a couple thousand, or so), has been a priority.

So, now I have two "SciFi" (and two Fantasy and two Currently_Reading, etc) collections and want to merge them, without copying the Kindle only books into Calibre.

Yes, poor original default choice. There is actually a Delete action along with Create in Customize. If you set the tags column to Delete and unchecked the ignore prefix (and made sure you put _ in the prefix or whatever character you used) and ran Preview/Create it should automatically delete any collection matching a tag name with a prefix exactly.

But merging is a bit tougher.

So, it sounds like I am stuck with two of each collection, until I manage to copy those books into Calibre, process them to fix titles, metadata, etc, assign the "new" collection in the custom column, then export them back to the Kindle (is this step necessary?) in order to add them to the new collections.

If you assign new values to the custom columns you can run Create without exporting the book to the Kindle again - the plugin reads the info from Calibre. But if you edit metadata in the book (title, etc) then you need to send to device again to have the book's data updated.

What I cannot do (if I understand correctly): Leave 200+ books in the Kindle collection, add the Calibre maintained books to it, import the collections and then create on the Kindle ... as this would remove those 200+ books from the collection, as they are not in Calibre, so would not be added to the collection when it is created. Correct?

You can always add any books manually to Kindle managed collections. You can use Edit Collections to do this. But if you imported the collection names then only books in Calibre would get a collection name in the custom column. So when you export the collection would be replaced with only Calibre books as Calibre doesn't know about the books not in Calibre.

I don't want to mix manual and automatic creation, so much as use automatic creation, have no duplicate collection names (other than the "_") and have all my books in collections (whether downloaded from Amazon or managed by Calibre).

And for this you need to have a set of collection names that are Kindle managed - and put all your non-Calibre books into it (plus any Calibre books you want in it as well), and a separate set of collection names that only contain Calibre books put there automatically by the plugin.

[meme]

Attached is a test version of the next release.

Can those of you who were having problems with non-ascii characters test this version. It may be working in 1.5.7, but I'm not sure it was using the 'right' approach so I want to test some newer code. Ideally you shouldn't notice any difference...

Can you run View report, and see if there are any warnings at the top about being unable to convert the path to utf-8. If so, for that book, can you try using Edit Collection to create a new collection and then add that book and save and Restart your Kindle to see if the book is actually in the collection. This will test if the pathname has to be converted to utf-8 before being put into a collection. (you can make a backup of your collections.json file or use Restore Collections if there are issues)

There will be a warning at the top of the View report that says 'System Encoding' . If you report issues, I'll need that value. Anyone who has something other than 'utf-8' or 'mbcs', can you let me know? NiLuJe - that probably means you

Can you check if any titles or authors with special non-ascii characters show up correctly in Edit Collections and View report.

And of course check if your other books are showing up in your collections after you do a Restart. If they don't seem to be, run Edit Collections, create a new collection and put a book in it, then save. Then Restart your Kindle and see if its in the Collection. (use Restore collections if you have issues, and you can always install 1.5.7 again to back out of this version after restoring your file).

The cache file will be rebuilt when you run this version so your first run may be slower than usual (added the book's encoding to the cache).


There are 2 areas with problems with regards to unicode/non-ascii characters. One is making sure that the pathname I use to create the hash code that goes into the collection file is the right type. Its not clear if the Kindle must use utf-8 or just unicode. At the moment I just convert the pathname to unicode without forcing it to utf-8 (its not clear if it converts to utf-8 or your local encoding by default) and do a hash on that. I need to see if I have to force utf-8, which might mean for some pathnames ignoring or replacing characters that can't be put into utf8. Whether just doing a simple unicode/hash works will depend on the system its running on - what encoding the system is - and what pathnames you have.

The second is an issue with titles/authors being encoded in Windows Latin-1 (CP1252) inside the books themselves. The CP1252 encoding is a special Windows (its always Windows, argghhh) which mostly maps to utf-8 but fails miserably with certain characters (a smart quote for example - character 0x92). This is what broke the plugin before - my sort algorithm on titles/authors would fall over when it tried to compare cp1252 and utf-8 strings when the cp1252 strings had a problem character in them. I've modified the code to identify the encoding used for a Mobi book and if it was encoded with cp1252 I decode it correctly. There are still some circumstances where the characters don't always show correctly, but generally it seems ok.

I've also re-written the general sorting algorithmn to be faster and less of a mess as I had a lot of unnecessary code duplicating what python could do for me. This should speed up view collections a little, but should be more noticeable in Edit collections when sorting the columns - but probably not a dramatic difference.

And I've added an About menu option so you can verify the version of the plugin.

EDIT - file removed, 7b available

[ikar]

I've tested this version and I had no problems with polish characters in titles and collections names. Everything seems to be ok.

[NiLuJe]

I'll try 1.5.7a ASAP, thanks

EDIT: Appart from the fact that the button's default action is always 'create', even when set to 'show menu' (Actually, it does both on my system, and sometimes manages to make calibre crash with a nifty segfault in QtCore ^^), no encoding issues on my end (iso-8859-15) (Checked View, Edit, Create, and everything is as it should on my Kindle after a framework restart ).

[meme]

1.5.7b - should fix the issue with Create running even when icon says Show menu

@ikar - thanks for the feedback

Originally Posted by NiLuJe

Appart from the fact that the button's default action is always 'create', even when set to 'show menu' (Actually, it does both on my system, and sometimes manages to make calibre crash with a nifty segfault in QtCore ^^), no encoding issues on my end (iso-8859-15) .

Thanks. Glad to see things are displaying/working ok. I'll be waiting for a few more checks, especially from those who recently had trouble with books in collections.

I thought I got the QtCore error, but it was pretty obvious what the bug was when I looked - I was calling Create even if you clicked Show menu The segfault is a result of trying to disconnect a trigger before its set - I needed to better test the current state. I couldn't reproduce the issue, but it was clear the code needed updating. Let me know if it works for you.

[EDIT: file removed ]

[scarlet slipper]

Just renamed some tags and deleted others and imported a couple of new books.

I recreated Collections and got a message telling me calibre had created 244 new collections. But I only have 199 tags. Did a restart and some of the tags I deleted or renamed from calibre are still showing up as a collection.

I don't know why I have more collections than I do tags and nor do I know why there are collections for which I have no current tags. Any suggestions for what I'm doing wrong would be appreciated.

[meme]

Originally Posted by scarlet slipper

I don't know why I have more collections than I do tags

Look in the report generated when you run Preview or Create (click Show Details on the window that pops up after it runs that tells you how many collections were created). All of the collections that are created are listed there - and which column in Calibre they came from, or if they were left on the Kindle and why.

If you deleted tags, then the plugin no longer knows that those tags/collection names are managed by Calibre - so it leaves them on the Kindle as Kindle-managed.

You have a couple of options to delete these old collection names/tags from the Kindle:

1. In Customization uncheck 'Preserve/Keep Kindle-only collections' and then run Preview to check, and Create to save. But if you have any collections that are only on the Kindle then you must put their names into the Never Delete box on the customization screen. Preview will tell you if it will delete or keep any specific collection.

2. Use Edit Collections and select the collections you don't want from the drop down menu and then click on the '-' (minus) button to delete the collection manually, then click save to update the Kindle.

[scarlet slipper]

<i>If you deleted tags, then the plugin no longer knows that those tags/collection names are managed by Calibre - so it leaves them on the Kindle as Kindle-managed.</i>

Ahh. Thanks for your suggestions, Meme. I didn't realize that. I figured since we have to recreate collections from scratch each time we update, the shuffling around would automatically occur.

[EbokJunkie]

Guys please, can anyone describe a procedure (if possible) to copy collections from one Kindle to another?
I have a defective Kindle that Amazon agreed to replace, I received the replacemnt today. Aall my books sit in folders and collections have respective names. No problem to copy folders, problem is in copying collections. I never used Calibre to manage my books and did everything manually.

Tried to save collections from old Kindle inside Calibre, then diconnected old Kindle, connected the new one...but saved collectiions seem to go away...
Is there any way to keep them and save to new Kindle?

[pchrist7]

Originally Posted by EbokJunkie

Guys please, can anyone describe a procedure (if possible) to copy collections from one Kindle to another?
I have a defective Kindle that Amazon agreed to replace, I received the replacemnt today. Aall my books sit in folders and collections have respective names. No problem to copy folders, problem is in copying collections. I never used Calibre to manage my books and did everything manually.

Tried to save collections from old Kindle inside Calibre, then diconnected old Kindle, connected the new one...but saved collectiions seem to go away...
Is there any way to keep them and save to new Kindle?

Hi there.
You don't say much about the contents on your old kindle. Only amazon books, mix, or ?
Look in the kindle user guide, search for "Adding Collections from another Kindle". This might work. Tried that in december when my first K3 got a screen error. I was not too happy, but then I have a mix of amaxon DRM books, and non DRM from many other sources.
The following could/might work for you, if the above doesn't, but I can't be sure:
NEW kindle: connect with USB, go to the systems folder, and make a backup copy of collections.json file. Disconnect from USB
OLD kindle: connect with USB, go to the systems folder, and copy collections.json to your desktop (or where ever ). Disconnect from USB
NEW Kindle: connect with USB, goto systems folder, and copy collections.json from your desktop to the systems folder.
Disconnect from USB, restart your new kindle, and it should work.

Maybe someone else has other experiences/suggestions..
A bit more info would be nice though

[meme]

The plugin only "sees" the collections stored on the Kindle currently connected. It does not copy them to your PC. So if you connect your new Kindle to the PC you will not see any collections on the new Kindle (because there aren't any until you create them).

As pchrist7 said you can simply copy your collections.json file from your old Kindle to your PC to make your own backup copy, then copy it to your new Kindle. Under Windows once you bring up File Manager and go to your Kindle, and add "system" to whichever drive letter is showing (e.g. "E:\system") to open the hidden "system" folder on the Kindle that contains the collections.json file. Obviously you need to restore all your files from your old Kindle to the new Kindle using exactly the same directories and filenames.

I can certainly see the need to backup/restore the collections file. If your Kindle were lost, you wouldn't have any collections.json file to restore, even if you could load your books again from Amazon or your PC. I haven't seen if Amazon provide any way to backup this file, but haven't looked. [EDIT: I see the Amazon User's guide entry mentioned above. Since I only have 1 Kindle I don't get the option to select collections from another device to load, but it sounds as if this would do a restore for you. Although I don't know if it stores any collection info for non-Amazon books on the Amazon site. If anyone can confirm it, then adding a backup facility wouldn't be needed).

I'll consider at some point whether the plugin should support making a backup copy of the collections file and the ability to restore it to any Kindle. At the moment customizations are saved per library and per Kindle on your PC, and backups are saved per Kindle on the Kindle. I'd need to save collections per Kindle on the PC. Probably save the collections automatically on connect and create/edit per Kindle - keeping either only the most recent one or the last few (to replace the current restore function), and offer a menu of choices for restore. The menu could just list dates of available backups per Kindle, possibly expandable to show collection names. But then it might need the ability to delete old Kindles, and if it allowed giving the Kindle a name to show in the list it would need to support deleting and renaming.

It goes from something simple to something that might be a bit more confusing, and may not be any more helpful than just copying the collections file along with the books. Of course, you need to know you have to copy the collections file, where it is and how to get it first.