User Interface plugins involving loops

[jackie_w]

I realise this is a long post but I hope at least someone will find the time to read and offer suggestions...

When writing a new User Interface plugin which involves looping around selected books or looping around files within a book container (or both) I have a decision to make. Should I:

1. Code it in the main QDialog as simple python for x in y: do_something.
   Easiest to code but may tie up calibre when running if each do_something is lengthy.
   
2. Code a standard QProgressDialog to loop around the do_something.
   Not too difficult to code. Ties up calibre when running but has the benefit of a Cancel button if the user gets impatient.
   
3. Code a QProgressDialog to set up a queue of thread jobs of do_something to run in the background.
   Quite challenging to code (for me, at least. I've only done it once.) Has the benefit of running in the background, but a potential headache of dealing with what to do if the user changes something whilst the do_somethings are running.

Can anyone offer any general advice on how to go about deciding which option to choose?

[jackie_w]

These are the specifics of my current "fun project" (audiobook creation) in case it makes any decision-making more obvious:

• The looping is around each HTML file in an epub/azw3/kepub container spine.
• The do_something is a 2-step process, neither of which modifies the book.
  • A call to Microsoft SAPI5 Speech interface to create a WAV file by TTS-speaking the container file text content.
  • Run a Windows EXE to convert WAV to MP3.
  Both of these processes could be quite lengthy (minutes not hours) if a book has chapters with a large wordcount.

As part of testing I've already coded Options 1 and 2 above with these observations:

Option 1:
I'm not sure if this is a problem or a feature

Having kicked off the 'Create MP3s' process via button-click it's not obvious how to cancel it. Closing the main dialog looks as if it closes the plugin but the WAV/MP3 creation process seems to keep going in the background. This may be a perfect solution - background running without the coding effort of thread/jobs - but I'm not sure of the technical implications.

Option 2:
Clicking the QProgressDialog Cancel button may happen during the WAV part or the MP3 part. The user wouldn't know which. ATMO the cancel doesn't take effect until both parts are complete. This could be quite a while after the Cancel button was actually clicked. It would be preferable if part 2 could be skipped if the Cancel button was clicked during part 1. I tried checking for self.wasCanceled() before starting part 2 but with no joy. I'm guessing I'm not understanding event processing well enough.

The QProgressDialog pop-up widget is completely empty until the completion of the first do_something cycle. This is a bit unnerving if it's a lengthy first cycle. Is there a way to get self.setLabelText and Cancel button to display immediately?

[kovidgoyal]

Your choices are:

1) Simply run your entire job in a thread hat produces its output in a temporary directory. Show a blocking progressdialog while this is happening. After the worker thread is complete, copy the output data into calibre on the main thread.

2) Use the calibre jobs infrastructure to do all the heavy lifting for you. This is how calibre itself runs conversion/bulk metadata download etc jobs. The idea is the same, you run the job to output to a temporary dir, when it is complete you bring the data into calibre. I'm sure there must besome existing plugins that use the jobs infrastructure you can copy from.

[jackie_w]

Originally Posted by kovidgoyal

I'm sure there must besome existing plugins that use the jobs infrastructure you can copy from.

Yes, I'm OK with this. It's what I did the only other time I tried the jobs approach. This is how I know it's challenging for me I guess it would be good practice to have a second attempt.

Is there some calibre help documentation I may have missed that would help me get a better overview of how it all fits together?

Originally Posted by kovidgoyal

Your choices are:

1) Simply run your entire job in a thread hat produces its output in a temporary directory. Show a blocking progressdialog while this is happening. After the worker thread is complete, copy the output data into calibre on the main thread.

2) Use the calibre jobs infrastructure to do all the heavy lifting for you. This is how calibre itself runs conversion/bulk metadata download etc jobs. The idea is the same, you run the job to output to a temporary dir, when it is complete you bring the data into calibre.

The jobs approach seems to be the option-for-all-seasons, it's just that it seems a bit OTT when the chosen book only contains a few short files.

Is there any issue with kicking off multiple MS API calls/EXEs? Might the OS grind to a halt?

The end product is a directory full of MP3s on a local disk. I'm not currently planning to import them back into calibre. Is there any need to hide them in a temp dir until the last step? The only output data I might want to bring back to the calibre GUI would be a text log of what was done and how long each cycle took to run. I think I should be able to sort something out for that.

[kovidgoyal]

Sadly no, I've never got around to documenting the jobs system APIs. Feel free to ask and I will try to help if you get stuck. You only need to use temp dirs if you are working on calibre data, as you must not access the calibre library folder in an asynchronous fashion (for the same reasons you cannot run multiple programs that touch the calibre library)

[jackie_w]

Well, I didn't get stuck after all, everything appears to be working OK However there were a few things I didn't fully understand. Perhaps you could enlighten me?

1. ParallelJob. This class has an argument called 'description'. What is this? I took a guess that it was an alphanumeric string that needed to be unique across all the individual queued jobs.
2. The argument 'timeout' in the method which does the actual work for each job. I have a method:
   

   def process_file(log, abort, myarg1, myarg2, timeout=30, in_process=True):

   What exactly is this timeout and how do you decide on a suitable value?
3. The .py file for the Microsoft Speech library I copied unchanged from an old TTS plugin for the calibre Viewer. It seems to be working OK as-is in my plugin. I'm not asking you to verify all 5000+ lines of it but please would you look at this short extract from the top of the file:
   
   Spoiler:

   # -*- coding: mbcs -*-
   # Created by makepy.py version 0.5.01
   # By python version 3.4.1 (v3.4.1:c0e311e010fc, May 18 2014, 10:38:22) [MSC v.1600 32 bit (Intel)]
   # From type library 'sapi.dll'
   # On Sun Apr  3 15:49:07 2016
   'Microsoft Speech Object Library'
   makepy_version = '0.5.01'
   python_version = 0x30401f0
   
   import win32com.client.CLSIDToClass, pythoncom, pywintypes
   import win32com.client.util
   from pywintypes import IID
   from win32com.client import Dispatch
   
   # The following 3 lines may need tweaking for the particular server
   # Candidates are pythoncom.Missing, .Empty and .ArgNotFound
   defaultNamedOptArg=pythoncom.Empty
   defaultNamedNotOptArg=pythoncom.Empty
   defaultUnnamedArg=pythoncom.Empty
   
   CLSID = IID('{C866CA3A-32F7-11D2-9602-00C04F8EE628}')
   MajorVersion = 5
   MinorVersion = 4
   LibraryFlags = 8
   LCID = 0x0

   
   There are a couple of references to Python 3 in there. I don't think these can be important because this library appears to work OK on my own PC which does not have any copy of Python 3. Is there anything there which rings any alarm bells? I've also attached the whole library file in case it's necessary.

[kovidgoyal]

1. Its a human readable description of the job that can appear in places like the status bar, jobs list, etc.

2. No idea, that's not from calibre code, is it from a plugin you copied?

3. If it works, dont worry about it. That header appears to imply the file was generated by something called makepy that was itself running on python3, but that may not mean that the generated file actually requires python 3

[jackie_w]

Originally Posted by kovidgoyal

2. No idea, that's not from calibre code, is it from a plugin you copied?

Yes, that's exactly what it is The perils of copying code from other people when you don't fully understand it. I used kiwidude's ExtractISBN plugin as my "guru" and a timeout didn't seem totally out-of-place in this context. I've just been back to check. I see that, although 'timeout' is an argument, the code which used it has been commented out. I'll get rid of it if it's not part of the jobs system.

This begs a follow-up question. The last arg of that same method is 'in_process' is this also not a part of calibre code?

Perhaps I need to examine a few more thread/jobs plugins to see what other redundancies I may be perpetuating. However, I think kiwidude probably wrote most of them so he may have a "house style".

[davidfor]

Originally Posted by jackie_w

Yes, that's exactly what it is The perils of copying code from other people when you don't fully understand it. I used kiwidude's ExtractISBN plugin as my "guru" and a timeout didn't seem totally out-of-place in this context. I've just been back to check. I see that, although 'timeout' is an argument, the code which used it has been commented out. I'll get rid of it if it's not part of the jobs system.

This begs a follow-up question. The last arg of that same method is 'in_process' is this also not a part of calibre code?

Perhaps I need to examine a few more thread/jobs plugins to see what other redundancies I may be perpetuating. However, I think kiwidude probably wrote most of them so he may have a "house style".

"in_process" isn't something that rings a bell for me from kiwidudes plugins, so I had a look at ExtractISBN. It is only used for an extra debugging statement. And it isn't really used. It defaults to True, but nothing that calls that function sets it to anything else.

[jackie_w]

Originally Posted by davidfor

"in_process" isn't something that rings a bell for me from kiwidudes plugins, so I had a look at ExtractISBN. It is only used for an extra debugging statement. And it isn't really used. It defaults to True, but nothing that calls that function sets it to anything else.

Thanks, David. I'd reached the same conclusion myself after a bit more digging, including checking out your Kobo Utilities plugin's use of the jobs system. It was remarkable how similar its comments were to the ones in ExtractISBN. Was one of kiwidude's plugins also your template for using the jobs system?

[davidfor]

Originally Posted by jackie_w

Thanks, David. I'd reached the same conclusion myself after a bit more digging, including checking out your Kobo Utilities plugin's use of the jobs system. It was remarkable how similar its comments were to the ones in ExtractISBN. Was one of kiwidude's plugins also your template for using the jobs system?

I pulled code from various plugins. Modify ePub was probably the main source, but I think I got the job stuff from something else. It might have been Count pages as I didn't need to update a book format but needed to update metadata. I think most of the plugins with jobs are based on one of these two.

And a lot of the comments have been left in the code as I didn't quite know what I was doing at the time I started the plugin. I really do need to do some cleanup.

[jackie_w]

Well, I posted this new TTS plugin in the Plugin forum but the call to the MS SAPI speech interface appears to fail if trying to run with calibre 32-bit ... sigh ... I really should have thought to check this before deciding to share.

I don't know enough to be able to fix this, if it even is fixable so I need to be able to fail gracefully, preferably during the initial plugin install, if that's possible.

I've set supported_platforms = ['windows'] in the plugin's __init__.py but I don't see anything else obvious in there for checking whether the PC and/or calibre is 64bit. I can see is64bit in calibre.constants, but where is the best place to do the test to cause the least user inconvenience if they don't have a suitable setup?

FWIW this was the error message:

Spoiler:

calibre, version 3.26.1
ERROR: Unhandled exception: <b>com_error</b>:(-2147352567, 'Exception occurred.', (0, None, None, None, 0, -2147200967), None)

calibre 3.26.1 Portable embedded-python: True is64bit: False
Windows-10-10.0.17134-SP0 Windows ('32bit', 'WindowsPE')
32bit process running on 64bit windows
('Windows', '10', '10.0.17134')
Python 2.7.12+
Windows: ('10', '10.0.17134', 'SP0', u'Multiprocessor Free')
Interface language: None
Successfully initialized third party plugins: EpubXCustomMetadata (0, 0, 2) && EpubModBuild (0, 1, 6) && EpubSplit (2, 4, 0) && KePub Metadata Reader (3, 0, 2) && Kobo Utilities (2, 6, 0) && Quality Check (1, 9, 11) && TTS to MP3 (0, 1, 0) && JS_Editor_Tools (0, 2, 5) && Count Pages (1, 8, 0) && Epub2Epub3 (0, 0, 16) && KePub Output (3, 0, 0) && KindleUnpack - The Plugin (0, 81, 5) && KePub Input (3, 0, 0) && EpubCheck (0, 1, 2) && ScrambleEbook (0, 0, 5) && Epub2Kepub (0, 0, 22) && KoboTouchExtended (3, 0, 2) && Extract ISBN (2, 0, 0) && Modify ePub (1, 3, 13) && KePub Metadata Writer (3, 0, 3)
Traceback (most recent call last):
  File "calibre_plugins.tts_to_mp3_plugin.uiaction", line 168, in show_dialog
  File "calibre_plugins.tts_to_mp3_plugin.tts_to_mp3", line 210, in __init__
  File "calibre_plugins.tts_to_mp3_plugin.tts_to_mp3", line 234, in initialise_data
  File "calibre_plugins.tts_to_mp3_plugin.other_dlgs", line 377, in get_voices_all
  File "calibre_plugins.tts_to_mp3_plugin.tts_typelib", line 2957, in GetVoices
  File "site-packages\win32com\client\__init__.py", line 459, in _ApplyTypes_
com_error: (-2147352567, 'Exception occurred.', (0, None, None, None, 0, -2147200967), None)

[kovidgoyal]

As far as I know, there is no way to do the test during installation, as yours is the only plugin I know of that cares about 32/64 bit (most plugins are pure python and so dont need to care). I suggest simply popping up the message when your plugin is clicked if you detect a 32bit environment.

[jackie_w]

Thanks.