Quote:
Originally Posted by stoduk
Hi szarroug3
As someone mentioned, there is an existing plugin (that mjwilson created, and I've been working on slowly in the background).
I thought I'd give your plugin a run to see how it compares, whether there is any point carrying on with both etc. Not really come to any conclusion on that - the Windows app is far and away the most stable and feature rich, both plugins have promise but have a lot of catching up to do.
[FWIW mine can create decent Xray files for me now, though I've really only tested with new format files. It doesn't yet create the Start/End/Author files, though I was going to add those soon]
Hit a few problems, thought I'd post them here. If you'd rather I submitted as issues on github I can do that. I've tested in two setups:
- OS X 10.10.5 with Calibre 2.56.0
- Windows 7 (under VirtualBox) with Calibre 2.56
Full gory details below. As it stands I haven't been able to get a working Xray using your plugin, but as it works for you I imagine I've got some small difference in my setup.
Cheers,
Anthony
1) Version 2.0.0 in Windows 7
"Selecting 'Book preferences" shows a dialog with only ASIN and shelfari URL (ie. missing the three buttons - search for ASIN, etc.)
2) Version 2.2.0
This is linked from github, but doesn't seem to exist!
2) Version 2.1.0 in Windows 7
a) Crash with book amazon likely doesn't know about
Spotted there was an updated version on the forum, tried this. Problem #1 now gone (fixed or hiding? You'll know more than me), but I get this error when running on Calibre's Quick Start Guide (both text boxes in dialog are empty, I assume because we are barfing trying to find the ASIN)
calibre, version 2.56.0
ERROR: Unhandled exception: <b>AttributeError</b>:'BookConfigWidget' object has no attribute '_asin_edit'
calibre 2.56 embedded-python: True is64bit: False
Windows-7-6.1.7601-SP1 Windows ('32bit', 'WindowsPE')
32bit process running on 64bit windows
('Windows', '7', '6.1.7601')
Python 2.7.9
Windows: ('7', '6.1.7601', 'SP1', 'Multiprocessor Free')
Successfully initialized third party plugins: X-Ray Creator
Traceback (most recent call last):
File "calibre_plugins.xray_creator.book_config", line 173, in search_for_asin
AttributeError: 'BookConfigWidget' object has no attribute '_asin_edit'
b) Lookup problems with A Christmas Carol/lack of error reporting
ASIN B00Q0LB318 is found (think that is coming from book as it is what I'm expecting), but the shelfari URL is 38316941. I thought things were failing here, but a little digging shows that this shelfari entry just has no details at all - so the "aliases" box is being populated with what it finds (ie. nothing!).
Some better status reporting here would go a long way - it isn't clear what happened (ie. shelfari page fetched but nothing found)
c) Fixing the shelfari URL to 4575254 (an entry with decent character data)
I edited the shelfari URL to use a better ID - but this caused Calibre to go in to extreme slow mode. A few seconds for each character press to be recognised. One time your plugin window closed without warning, the second time it worked - and I get a list of Characters to edit!
[Note: I tried to paste the URL in once, instead of typing, and that caused Calibre to lock up completely. Assume this is a similar problem]
Is your plugin doing anything while the "Shelfari URL" text box is being edited? I wonder if edit_shelfari_url() is being run with each character press - if so that could be sllllloooowww. Please don't do that, just update once the click button has been pressed (or after a few moments of inactivity) 
d) Making an xray finally
The x-ray that is generated (note: I didn't edit aliases, so we've just got the default character names), and I get garbage. I'm sure Peter is a nice guy, but he really isn't a main character, and certainly he isn't mentioned 65870 times (compared to zero for everyone else!). [I'm sure when I ran this yesterday I got the same result, but with Fred getting all the action]
sqlite> select id, label, count from entity;
id label count
---------- ---------- ----------
0
2 Jacob Marl 0
3 Bob Cratch 0
4 Ghost of C 0
5 The Ghost 0
6 The Ghost 0
7 Mrs. Cratc 0
8 Tiny Tim 0
9 Martha Cra 0
10 Peter Crat 65870
11 Belinda Cr 0
12 Mr. Fezziw 0
13 Mrs. Fezzi 0
14 Ebenezer S 0
15 Dick Wilki 0
16 Fred 0
17 Jacob Marl 0
18 Joe Miller 0
19 Caroline 0
20 Topper 0
21 Belle 0
22 Fan 0
23 Almshouse 0
24 Blind man' 0
25 Humbug 0
sqlite>
e) editing aliases
I tried adding a single alias ("Scrooge" for the main character) and that had no effect - still Peter Cratchit gets all the action, as in previous case.
3) Latest github version in OS X
a) unmodified
I cloned your github and tried that out in OS X - it doesn't work, as I see mentioned in the forum. It would be a shame to not support all OS, especially when the current problem is so minor. You just need to figure out a way to look for external devices in OS X/linux - I suspect something with 'mount' would work.
b) hacked to workaround problem
4) general comments
a) error reporting
you seem to be swallowing some errors, instead of printing the original python stacktrace. eg. I think I got the "windll" assertion with the 2.0.0 version of the plugin, but I get something more cryptic with the github version (KeyError for 'send_status'). That might make debugging harder..
b) interacting with calibre DB
I had been putting this step off, as I hadn't figured out how to do it. I *think* what you are doing is considered naughty, you shouldn't be modifying the Calibre DB under the covers. I meant to start a thread on how to fix that, hopefully Kovid or other has some ideas.
[I think if a book gets renamed or deleted, you'll be leaving junk scattered around the filesystem. Not a terminal problem, but something to be aware of]
c) interacting with kindle
As with (b) I hadn't figured this out. I wondered if Kindle driver would need to be modified. Again, I was going to ask the experts.
[what you are currently doing is duplicating code from Kindle driver, but it should be harmless as you'll only be doing what I do by hand currently]
d) modifying ASIN in books directly
If I understand your code you are modifying the MOBI on the Kindle device directly to update the ASIN. This seems odd! FWIW the "Quality Check" plugin can update ASIN as needed, so might be better to just call that to fix books in the Calibre DB directly. |
Anthony,
First, thanks for the review! It's super helpful.
1) Version 2.0.0 in Windows 7
"Selecting 'Book preferences" shows a dialog with only ASIN and shelfari URL (ie. missing the three buttons - search for ASIN, etc.)
- Version 2.0.0 doesn't have the 3 buttons. I added that in 2.2.0 (which I'll talk about in #2)
2) Version 2.2.0
This is linked from github, but doesn't seem to exist!
- Thanks for catching this. I must've forgotten to push the file to github :/
2) Version 2.1.0 in Windows 7
a) Crash with book amazon likely doesn't know about
Spotted there was an updated version on the forum, tried this. Problem #1 now gone (fixed or hiding? You'll know more than me), but I get this error when running on Calibre's Quick Start Guide (both text boxes in dialog are empty, I assume because we are barfing trying to find the ASIN)
calibre, version 2.56.0
ERROR: Unhandled exception: <b>AttributeError</b>:'BookConfigWidget' object has no attribute '_asin_edit'
calibre 2.56 embedded-python: True is64bit: False
Windows-7-6.1.7601-SP1 Windows ('32bit', 'WindowsPE')
32bit process running on 64bit windows
('Windows', '7', '6.1.7601')
Python 2.7.9
Windows: ('7', '6.1.7601', 'SP1', 'Multiprocessor Free')
Successfully initialized third party plugins: X-Ray Creator
Traceback (most recent call last):
File "calibre_plugins.xray_creator.book_config", line 173, in search_for_asin
AttributeError: 'BookConfigWidget' object has no attribute '_asin_edit'
- I'll have to look into this. I haven't run across that in 2.1.0 but I'll try it.
Update: turns out I had a typo in the file. Fixed it
b) Lookup problems with A Christmas Carol/lack of error reporting
ASIN B00Q0LB318 is found (think that is coming from book as it is what I'm expecting), but the shelfari URL is 38316941. I thought things were failing here, but a little digging shows that this shelfari entry just has no details at all - so the "aliases" box is being populated with what it finds (ie. nothing!).
Some better status reporting here would go a long way - it isn't clear what happened (ie. shelfari page fetched but nothing found)
- Yes, I've been really bad about status reporting in the book preferences. I'll have to add a status message to these.
c) Fixing the shelfari URL to 4575254 (an entry with decent character data)
I edited the shelfari URL to use a better ID - but this caused Calibre to go in to extreme slow mode. A few seconds for each character press to be recognised. One time your plugin window closed without warning, the second time it worked - and I get a list of Characters to edit!
[Note: I tried to paste the URL in once, instead of typing, and that caused Calibre to lock up completely. Assume this is a similar problem]
Is your plugin doing anything while the "Shelfari URL" text box is being edited? I wonder if edit_shelfari_url() is being run with each character press - if so that could be sllllloooowww. Please don't do that, just update once the click button has been pressed (or after a few moments of inactivity)
- Yeah, I've been making it check everytime the box is edited which is bad on my behalf. Will update.
d) Making an xray finally
The x-ray that is generated (note: I didn't edit aliases, so we've just got the default character names), and I get garbage. I'm sure Peter is a nice guy, but he really isn't a main character, and certainly he isn't mentioned 65870 times (compared to zero for everyone else!). [I'm sure when I ran this yesterday I got the same result, but with Fred getting all the action]
sqlite> select id, label, count from entity;
id label count
---------- ---------- ----------
0
2 Jacob Marl 0
3 Bob Cratch 0
4 Ghost of C 0
5 The Ghost 0
6 The Ghost 0
7 Mrs. Cratc 0
8 Tiny Tim 0
9 Martha Cra 0
10 Peter Crat 65870
11 Belinda Cr 0
12 Mr. Fezziw 0
13 Mrs. Fezzi 0
14 Ebenezer S 0
15 Dick Wilki 0
16 Fred 0
17 Jacob Marl 0
18 Joe Miller 0
19 Caroline 0
20 Topper 0
21 Belle 0
22 Fan 0
23 Almshouse 0
24 Blind man' 0
25 Humbug 0
sqlite>
- Can you send me the book to try this out with. I could've sworn I got the book parsing down. It works for all my books but maybe I missed something. Or I messed something when making changes for the aliases.
e) editing aliases
I tried adding a single alias ("Scrooge" for the main character) and that had no effect - still Peter Cratchit gets all the action, as in previous case.
- look at d
3) Latest github version in OS X
a) unmodified
I cloned your github and tried that out in OS X - it doesn't work, as I see mentioned in the forum. It would be a shame to not support all OS, especially when the current problem is so minor. You just need to figure out a way to look for external devices in OS X/linux - I suspect something with 'mount' would work.
- Yes, I haven't added support for OS X and Linux. I'm the only dev on this project and I don't have access to either system at the moment. I'm looking into it but this will take some time to add.
b) hacked to workaround problem
4) general comments
a) error reporting
you seem to be swallowing some errors, instead of printing the original python stacktrace. eg. I think I got the "windll" assertion with the 2.0.0 version of the plugin, but I get something more cryptic with the github version (KeyError for 'send_status'). That might make debugging harder..
- Still working on catching all the bugs but I feel like the user doesn't really need to see all the exceptions thrown so I print prettier error messages. Example, if the amazon page isn't found, instead of throwing the httplib exception thrown, I print Amazon page not found. You can see this in the job details.
b) interacting with calibre DB
I had been putting this step off, as I hadn't figured out how to do it. I *think* what you are doing is considered naughty, you shouldn't be modifying the Calibre DB under the covers. I meant to start a thread on how to fix that, hopefully Kovid or other has some ideas.
[I think if a book gets renamed or deleted, you'll be leaving junk scattered around the filesystem. Not a terminal problem, but something to be aware of]
- As far as I can see, editing the calibre db isn't bad. I'm using Calibre's functions to do it so I'm not opening the DB and just modifying directly or anything. See here:
https://manual.calibre-ebook.com/gen...l#set-metadata
c) interacting with kindle
As with (b) I hadn't figured this out. I wondered if Kindle driver would need to be modified. Again, I was going to ask the experts.
[what you are currently doing is duplicating code from Kindle driver, but it should be harmless as you'll only be doing what I do by hand currently]
- Yeah I wasn't sure how to do it using Calibre's built in functions. I end up doing it manually (which also explains why I can't support OSX and Linux just yet.. I have no idea how to get to the kindle)
d) modifying ASIN in books directly
If I understand your code you are modifying the MOBI on the Kindle device directly to update the ASIN. This seems odd! FWIW the "Quality Check" plugin can update ASIN as needed, so might be better to just call that to fix books in the Calibre DB directly.
- This was actually quite a complicated decision I made. I wanted to be able to support multiple book formats. Calibre allows you to put multiple formats of the book on your device if you want to. If there's a book with ASIN 123 and it has both azw3 and mobi on the device, the sdr folder for both books will be exactly the same meaning I would have to put 2 x-ray files in the sdr with the same exact name which obviously isn't possible. To get around this, I update the asin on the device to add the format to the end effectively allowing both books to have an x-ray. I don't why anyone would want to have more than one format of a book on a kindle but that's not for me to decide.
All of that being said, I did see your version a few days ago and I saw that your github page states it's not fully functional. Maybe we can merge projects or something. I would definitely love all the help I can get coding (especially considering my issue with other OS'), responding to bug reports and feature requests, and making design decisions. Like I said, I'm the only dev on this project. I did have some help from the other author listed but he helped me with figuring out how to do certain things rather coding. There is a lot of clean up I need to do which is probably what I'll be doing for the next version. PM me if you're interested in merging or something.