FTS for UTF-8 Text Formats in Non-TXT File Extensions

[DaltonST]

Kovid,

I would like FTS to index RIS "Citation" file formats, which are simply UTF-8 Text in a Non-TXT File Extension/format in Calibre, but that have a rigidly defined content structure via RIS Tags (e.g. AB - "Abstract") for which there is a great deal of functionality in Calibre now via 3 different plugins.

I have not been able to determine any method to do so other than writing a new InputFormatPlugin (and a new OutputFormatPlugin) to "convert" UTF-8 Text RIS into UTF-8 Text TXT, saving a .ris rather than a .txt.

Is there another way I haven't found, such as an internal dict in an already-existing Calibre module that relates an input format to its equivalent input and output format which could be updated by an existing plugin, such as: {'RIS':'TXT'} ?

Or by MIME Type? {'RIS':'application/x-research-info-systems'} and {'application/x-research-info-systems':'text/plain'}.

Or should I write 2 new conversion plugins just to do the above?

Thanks for your help.

DaltonST

[kovidgoyal]

You need an input format plugin not an output one. FTS requires a way to normalize a book to a common HTML format which it can scan.

[DaltonST]

Deleted.

[chaley]

Originally Posted by DaltonST

Since all of the existing input conversion plugins are "built in" to Calibre, they don't look like the documentation, so they were of no help.

There are several InputFormatPlugins in the index of plugins:

• KFX Input
  Conversion input plugin to convert books in the Amazon KFX format.
• Language Cleaner
  A conversion plugin that filters profanity and replaces vulgar language.
• DOC Input
  Process input microsoft doc files.
• KePub Input
  Allow calibre to convert KePub format files to other formats.

[DaltonST]

Chaley,

Thanks. As the 'Language Cleaner' demonstrates, the Class must be in the __init__.py and not in its own .py file.

No wonder the "built-ins" were no help; they are not in a .zip file for me to autopsy.


DaltonST

[DaltonST]

Debugging works easily and reliably in both classes of plugins I have written, GUI and FileType. Except, now, an input conversion plugin. I cannot get a single debugging line out.

The Documentation for its particular plugin Class says "use the log". I have, in so many ways, and nothing ever happens.

After executing calibre-customize -b as normal, I then have tried three ways that I know of to run in debug mode:
calibre-debug -g
calibre-debug --gui-debug="C:\Users\xxx\Desktop\debug.log"
calibre.exe, then restart in debug mode using the menu option to do so.

None has ever worked, except that it shows that the FTS scanner fails because its (FTS') input conversion plugin that I am trying to develop and test is failing. Given that I have never seen a single debug line, or any other line, printed or logged successfully, I would be shocked if it were successful.

I have tried using the 'log' argument passed from whomever is calling the plugin that I cannot debug, including: log.debug; log.message; and log.error.

Results: Nada. Nichts. Nothing.

I tried using my own Python logger via logging.getLogger(), etc., but still nothing.

I tried writing out the accumulated debugging text to a .txt file using fd.write(), but nothing appeared in its destination folder, and since I have zero debugging output (actually, any output of any type), I have no clue as to why. And I never will without debugging.

So, might someone please graciously tell me how to output a debugging log from an input conversion plugin?


Thank you.


DaltonST

[chaley]

Using this trivial plugin:

from calibre.customize.conversion import InputFormatPlugin

class TestInput(InputFormatPlugin):
    name = "TestIt Input"
    author = "me"
    version = (1, 0, 0)
    minimum_calibre_version = (2, 0, 0)
    supported_platforms = ["windows", "osx", "linux"]
    description = "Test logging"
    file_types = {"foo",}

    def __init__(self, *args, **kwargs):
        InputFormatPlugin.__init__(self, *args, **kwargs)
        print('here in TestInput')

    def convert(self, stream, options, file_ext, log, accelerators):
        print('Hi There printed')
        log.error("Hi There tp log.error")
        raise ValueError('my exception')

I get this output when I attempt to convert a .FOO document to EPUB using the convert button.

Spoiler:

Convert book 1 of 1 (doc)
here in TestInput
Conversion options changed from defaults:
read_metadata_from_opf: 'C:\\Users\\Charles\\AppData\\Local\\Temp\\calibre _2i3y7vaf\\x94g4ylx.opf'
verbose: 2
output_profile: 'tablet'
Resolved conversion options
calibre version: 6.14.0
{'asciiize': False,
***CLIP***
}
InputFormatPlugin: TestIt Input running
on C:\Users\Charles\AppData\Local\Temp\calibre_2i3y7v af\e7fujkl_.foo
Hi There printed
Hi There tp log.error
Traceback (most recent call last):
File "runpy.py", line 196, in _run_module_as_main
File "runpy.py", line 86, in _run_code
File "site.py", line 83, in <module>
File "site.py", line 78, in main
File "site.py", line 50, in run_entry_point
File "calibre\utils\ipc\worker.py", line 215, in main
File "calibre\gui2\convert\gui_conversion.py", line 38, in gui_convert_override
File "calibre\gui2\convert\gui_conversion.py", line 25, in gui_convert
File "calibre\ebooks\conversion\plumber.py", line 1108, in run
File "calibre\customize\conversion.py", line 242, in __call__
File "calibre_plugins.TestInput.__init__", line 20, in convert
ValueError: my exception

Both the print and the log.error() messages are there.

The call on import by FTS to get the text does seem to throw away the logs, but you can debug the plugin using convert.

[DaltonST]

You solved it. The FTS Scanner is eating the log and print.

Running a conversion works.

The online documentation should have a "how to debug" section for input conversion plugins to avoid future wasted effort and time figuring it out.

Thank you, Chaley.



DaltonST