[GUI Plugin] Kindle Collections (old)

[meme]

Ok, here's another update

Rie142, CWatkinsNash - this might fix your issues, please give it a try. Thanks for asking about this - I really need people to report any issues with their specific environments as I'm more than happy to work on fixing them.

purple mink - this includes yet another check for invalid files - can you try it again?

seaquay - 0.7.45 looks like it fixed the issue with templates in columns, so try the latest plugin

CWatkinsNash - what can I say, but thanks for a thorough test and explanation. Its great to have the support of others!

The plugin was reading and using all the metadata.calibre data. But Calibre saves information on every book on the Kindle - even books no longer in Calibre. But it does flag which ones are in Calibre and which aren't. I was parsing all of them, until now. Version 1.3.3 should skip over any tags/columns in books that aren't in Calibre, which it should have done from the beginning.


There might be a couple bugs in Calibre that I'll follow up. If you delete a book, the metadata.calibre file is not updated to make the application_id null so the book appears to still be in Calibre if you read the file. And if you delete a book and reconnect the device, the application_id is correctly null, but the tag information, etc. is not cleared.

[kiwidude]

Here's my feedback on 1.3.2 after I at last had a chance to try this myself.

I had a fair number of issues but I think it is very much the case that many were not the fault of the plugin but more all the "layers" of complexity that coexist in Calibre. When you consider all the variables of book metadata, send to device template, metadata plugboard template and Kindle device plugin configuration sitting underneath, this plugin has to balance very precariously on top of the lot. I am not surprised that others have had issues, and I am sure an awful lot more will in future. Again I repeat that none of the major issues I had had were the fault of the plugin at all, nor were they bugs in Calibre. It was just ignorance on my part on knowing exactly how to get all the parts in sync for it to work.

So what did I have go wrong?
- I changed the "Send to device" template, as I was previously using a different collection manager which relied on specific names of folders to generate collections. However while I changed this in Preferences -> Sending books to devices, there is also another value I discovered stored in Preferences -> Plugins -> Device Interface Plugins -> Kindle 2/3 Device Interface. In the meantime I had doubled up books etc.
- When I first created collections I had more collections than I "expected". However I traced this down to a metadata plugboard override I had previously setup for authors, to shorten the author name to surname and initial. So the plugin created collections created for both the full author name and the shortened name.

In the end to get it working I decided to rip out that metadata plugboard override of author, delete all the books on my Kindle, restarted then resent all the books to the kindle, then ran the collection manager again. With all the ducks lined up it did exactly what was advertised which is great.

Does anyone else find it takes several minutes before Calibre starts responding again when they connect or eject? (Even with only a handful of books on the Kindle)?

My couple of plugin specific things I noted:
(1) If you go into the configuration screen for the plugin when the Kindle is not connected I note it doesnt list anything in the grid. You can change (or seem to) the checkboxes at the bottom of the dialog. Left me a bit confused and concerned about making changes. If you can't configure the grid then some sort of visual messages about requiring the Kindle to be connected to view and change would help. It just looked like it had lost my settings when you see an empty grid.
(2) When I deleted all collections on my Kindle, restarted then rang the plugin I got several "error" messages about how no json file was found in two different dialogs lpus the report. Seeing a file not found in an error dialog makes me think it has/will fail - when in fact it just means you have no collections on your Kindle to start with. As this appears to be informational rather than terminal/error I think it shouldn't be thrown in the user's face quite so much or at least explain in a less "critical" fashion.

Much credit and thanks to meme for the plugin. Once I eventually got sorted I now have exactly what I want without having to run external tools against the Kindle.

[meme]

1.3.3 might have helped not show books you didn't expect. At least the preview can tell you when things might double up. Difficult to track down issues with lots of customized templates, etc. - thanks for testing them out.

It does take a few seconds for Calibre to recognize connect/disconnect since its not checking every second. Makes debugging fun - which is why I allow it to run against a directory copy of the Kindle when testing at least some parts

1. You should get a warning if the Kindle is not detected, or if there is no metadata.calibre file already. It may be that you are running the plugin after disconnecting in a way or at a time I didn't check for. I'll need to retest all the scenarios, but if you remember what you did, let me know.

2. I'll check into the collections.json message. I made it an error in case there was a problem with the connection - but its much more likely that its a first time collection creation - and creating it if it isn't there isn't going to hurt anything.

Great stuff - thanks again for all the help and encouragement

[Rie142]

Originally Posted by ElizabethN

Some amazon books come with tags preloaded; is there a chance that the plugin is picking those up?

if you're in device library on calibre, highlight a book and do a right click (on a windows mouse) to get to the context menu, the option to add a book to library will be there. it may be worth checking to see if any of those unwanted tags came from that source.

when added to the main library, associated tags from amazon should show up as well in calibre's tag field.

Thanks, I am wondering if that is where the unwanted tags are coming from. Now I know how to check. Someday I will get this all working for me.

It is harder to teach an Older dog new tricks but I am trying to learn.

[CWatkinsNash]

Originally Posted by meme

Ok, here's another update

Rie142, CWatkinsNash - this might fix your issues, please give it a try. Thanks for asking about this - I really need people to report any issues with their specific environments as I'm more than happy to work on fixing them.

Yes, yes it did fix my issues. Thank you!! So much happiness in that little zip file. And I must say, the preview feature, though it's been here for a few iterations of the plugin, is brilliant. It's obviously good so people can see changes before they commit to them, but it's also a "Debug Lite". Very helpful.

Originally Posted by meme

CWatkinsNash - what can I say, but thanks for a thorough test and explanation. Its great to have the support of others!

You're quite welcome. I'm happy to offer what little help I can give.

Originally Posted by meme

The plugin was reading and using all the metadata.calibre data. But Calibre saves information on every book on the Kindle - even books no longer in Calibre. But it does flag which ones are in Calibre and which aren't. I was parsing all of them, until now. Version 1.3.3 should skip over any tags/columns in books that aren't in Calibre, which it should have done from the beginning.

That explains everything perfectly and it is now working as you describe. Again, thank you!

Originally Posted by kiwidude

Does anyone else find it takes several minutes before Calibre starts responding again when they connect or eject? (Even with only a handful of books on the Kindle)?

Yes, and it seems more extreme than it is because of the way it runs separate processes. The GUI takes a backseat while the jobs start running and there's no immediate visual feedback about what's going on. When I first started using calibre, I had to train myself to leave it alone till it becomes responsive again. It really does look like it's crashing sometimes, but I've accepted that it wants to be left alone when it's busy. I can relate.

[Rie142]

Originally Posted by meme

Ok, here's another update

Rie142, CWatkinsNash - this might fix your issues, please give it a try. Thanks for asking about this - I really need people to report any issues with their specific environments as I'm more than happy to work on fixing them.

Looks like it works better for me now. I ran it in the debug mode. Now I would like to know where the collections.json file is located at. I have looked all over on my computer and on the Kindle. What am I missing. Can you help me find it so I can get rid of all the information of old books that it is holding? I am still getting a couple odd tags but not as bad as before.

[CWatkinsNash]

Originally Posted by Rie142

Looks like it works better for me now. I ran it in the debug mode. Now I would like to know where the collections.json file is located at. I have looked all over on my computer and on the Kindle. What am I missing. Can you help me find it so I can get rid of all the information of old books that it is holding? I am still getting a couple odd tags but not as bad as before.

It's in the "system" folder, which is hidden by default. In Windows, you can either change your folder preferences in Explorer to show hidden folders ("Organize" button in Vista>Folder and Search Options>View tab), or you can open the Kindle "drive" in Explorer, go up to the address bar, and simply add system\. For example, my Kindle is f:\. In Vista (and I guess Win7) it looks like you can't type in there but you can - just click in empty space in the Explorer address bar and it will change from "buttons" to a field you can type in. Screenshots attached.

[meme]

At least on Vista, you also need to set it to show system folders since the 'system' folder is being seen as a system folder to Windows.

Rie142 - for the couple of collections you get that you don't expect - what does it say about them in the report that is displayed when you run Preview? It should indicate if they are from Calibre or on the Kindle already. If they are on Kindle already you can just use your Kindle to manually delete it after making sure you don't want it.

[kiwidude]

Originally Posted by meme

1.3.3 might have helped not show books you didn't expect. At least the preview can tell you when things might double up. Difficult to track down issues with lots of customized templates, etc. - thanks for testing them out.

The preview is great in that it tells you what it is "about" to do, so you don't get surprised after rebooting your Kindle. However of course it doesn't currently help you identify "why" that collection would get created in terms of exactly what circumstance caused it to be and what you should do to remove it . Maybe over time the plugin might be able to offer a little more in that regard at the bottom of the report or something, or maybe that is too hard and you just need a list of "things to check" somewhere.

1. You should get a warning if the Kindle is not detected, or if there is no metadata.calibre file already. It may be that you are running the plugin after disconnecting in a way or at a time I didn't check for. I'll need to retest all the scenarios, but if you remember what you did, let me know.

Ahhh, so I shouldn't have been able to see it . I am running Windows 7. I had the Kindle connected, did my stuff with your plugin, disconnected the Kindle and then a little later through Preferences->Plugins brought up the configuration dialog.

I just tried again on your latest version 1.3.3 - except this time without connecting a Kindle. I now get this error:

calibre, version 0.7.45
ERROR: Unhandled exception: <b>KeyError</b>:'ignore_all'

Traceback (most recent call last):
  File "D:\Dev\CalibreDev\latest\calibre\src\calibre\gui2\preferences\plugins.py", line 257, in double_clicked
  File "D:\Dev\CalibreDev\latest\calibre\src\calibre\gui2\preferences\plugins.py", line 330, in modify_plugin
  File "D:\Dev\CalibreDev\latest\calibre\src\calibre\customize\__init__.py", line 132, in do_user_config
  File "<string>", line 79, in config_widget
  File "C:\Users\Grant\AppData\Roaming\calibre\plugins\Create Kindle Collections.zip\create_kindle_collections_code.py", line 1371, in config_widget
KeyError: 'ignore_all'

[meme]

Yep, I'm rewriting the check for Kindle connected and the customization pre-checks to be a lot more thorough and correct. Unfortunately when I made the error dialogs nicer for the Menu customization entry, it broke the customization invoked by the Preferences dialog. (See the separate thread about do_user_config - I was trying to avoid the large window created when there wasn't anything to customize, but only fixed it in one area).

I do have some more information on where collections come from in the debug output - but I might be able to append it to the report. Debug shouldn't need to be run unless there is an unexpected issue.

[Rie142]

Originally Posted by meme

At least on Vista, you also need to set it to show system folders since the 'system' folder is being seen as a system folder to Windows.

Rie142 - for the couple of collections you get that you don't expect - what does it say about them in the report that is displayed when you run Preview? It should indicate if they are from Calibre or on the Kindle already. If they are on Kindle already you can just use your Kindle to manually delete it after making sure you don't want it.

Thanks, I figured it out that the collections I was getting extra were already on my kindle. Everything is good now. Just a few books that aren't in collections for some reason. Probably just a miss placed tag. Won't take me long to clean up.

Thank you so much for all the help. This old dog is slowly learning how to do things. Just need to read slower so I don't miss things.

[Rie142]

Originally Posted by CWatkinsNash

It's in the "system" folder, which is hidden by default. In Windows, you can either change your folder preferences in Explorer to show hidden folders ("Organize" button in Vista>Folder and Search Options>View tab), or you can open the Kindle "drive" in Explorer, go up to the address bar, and simply add system\. For example, my Kindle is f:\. In Vista (and I guess Win7) it looks like you can't type in there but you can - just click in empty space in the Explorer address bar and it will change from "buttons" to a field you can type in. Screenshots attached.

Ok I am off to try this. Yes I am running Vista. Seems to work fine as an OS for me. Now just need to find where it has the hidden folders. I will probably just add it to the explorer bar. I will make it work.

[seaquay]

@meme, the custom column fix worked. Thank you.

There was another minor hiccup that I had mentioned in the earlier mail

quote...I changed the column from 'kindle' to 'kindle_col' and both can be seen when customising the plugin in the plug-in GUI. Maybe there is a bug here....unquote

It is not creating an issue, but I wondered just for the sake of completeness

[meme]

Originally Posted by seaquay

...I changed the column from 'kindle' to 'kindle_col' and both can be seen when customising the plugin in the plug-in GUI. Maybe there is a bug here....

I suspect this is because you must eject the Kindle and reconnect it any time you add/update/delete any books or columns. I'll need to emphasize this more. Although I mentioned Calibre might have some bugs in the metadata.calibre writing, its actually working as expected. We just have to be careful about making sure the Kindle's metadata.calibre file is always up to date - and the best way is to disconnect and reconnect the Kindle whenever you make any changes to Calibre.

Thanks for mentioning it - if it can be fixed, I'd like to fix it

[purple mink]

Originally Posted by meme

Ok, here's another update

Rie142, CWatkinsNash - this might fix your issues, please give it a try. Thanks for asking about this - I really need people to report any issues with their specific environments as I'm more than happy to work on fixing them.

purple mink - this includes yet another check for invalid files - can you try it again?

seaquay - 0.7.45 looks like it fixed the issue with templates in columns, so try the latest plugin

CWatkinsNash - what can I say, but thanks for a thorough test and explanation. Its great to have the support of others!

Ok thanks for helping us out. I ran 1.3.2 once again and it seemed to be alright. Gotta read up more on this plugin mind you, now that it's working for me. I will also try out 1.3.3