Preprocess txt and txtz

[draganHR]

Hi, i am working on plugin and i need help.

My plugin should modify markdown formatted txt files before converting to other formats. Currently i'm using `FileTypePlugin` with on_preprocess = True. Inside run method i read current txt file, process it and save as new temp file.

Is this right way to do this, or is there better/easier/correct way?

I would like to do the same with txtz files, any advices? Do i need to extract it and zip content after processing manually?

Thanks

[kovidgoyal]

That is the right way, and yes for txtz you will have to rezip the file into a new txtz file.

[Agama]

Originally Posted by draganHR

Hi, i am working on plugin and i need help.
My plugin should modify markdown formatted txt files before converting to other formats.

Welcome to Mobile Read!

Your plugin sounds very interesting. I pre-process markdown txt files prior to conversion but have resorted to using VBScripts because I already know VB and not Python. This means a lack of integration with calibre which is a bit of a downside. Do you have any plans to make your plugin publicly available? I may be able to use it as a starting point for my own plugin!

[draganHR]

Thanks for answers, guys.

@Agama, i will publish my plugin soon, i will PM you when i finish it so you can take a peek at source code.

[draganHR]

I have made some progress, my plugin now supports both txt and txtz files.
I still have some followup questions:

• Is name of text file inside txtz archive always index.txt?
• Whenever i convert a book, `run()` method is called twice, any ideas why? (only `on_preprocess` is `True`)

Current version is in attachment.
Thanks.

(Just quick info if anyone is interested, this plugin is used to manage footnotes and endnotes in markdown formatted text files, i will make post about it when plugin is finished)

[kovidgoyal]

IIRC its index.text for textile and index.txt for others. Use

import traceback
traceback.print_stack()

to see why it is being called twice

[Agama]

Just quick info if anyone is interested, this plugin is used to manage footnotes and endnotes in markdown formatted text files, i will make post about it when plugin is finished.

I simply use markdown to manage my footnotes, (including reverse links from the footnote back to the main text), so it will be interesting to see what you are doing and how it enhances basic markdown.

[Agama]

I've had a play with your plugin using your sample file and it seems to work nicely. I'll test it further with my own text file. Looking at the source code, there are various print statements. Where can I see the output from these when the plugin runs?

One minor suggestion: How about allowing the special character to be user-defined and saved as part of the plugin, rather than hard-coding the dagger character?

[Agama]

I'm having trouble getting your plugin to work with my own simple test file, (attached with resultant ePub), - perhaps I am missing something obvious. It works with your test file so I know that the plugin is getting called during conversion, and it does make foot noting easy. Any ideas what I'm doing wrong?

[draganHR]

Originally Posted by Agama

Looking at the source code, there are various print statements. Where can I see the output from these when the plugin runs?

"Show job details" button in Jobs list.

Originally Posted by Agama

I'm having trouble getting your plugin to work with my own simple test file, (attached with resultant ePub), - perhaps I am missing something obvious. It works with your test file so I know that the plugin is getting called during conversion, and it does make foot noting easy. Any ideas what I'm doing wrong?

There was encoding problem - my file was utf-8 and your wasn't. Try updated version pls.

[Agama]

The updated version worked, so I've now changed some of the links to inline references, but these didn't work. Files attached as before.

By the way what does utf-8 mean and do I need to make files in this format? I simply opened Notepad++ on a Windows 7 platform and typed the document. I've never known calibre have any problems with any other documents that I have typed.

The Job Details button worked just as you pointed out and looks a very handy option for debugging plugins.

(I'm learning lots from your code and reckon I will be able to tackle my plugin soon. It's a bit obscure but will take guitar tablature exported as plain text from TablEdit and apply Markdown to enable clean conversion to ePub in calibre. Then I can use calibre's library to catalogue all my Tabs. Just a shame that my Sony reader cannot play MIDI files so that I can hear the Tab whilst reading it!)

[user_none]

Originally Posted by Agama

By the way what does utf-8 mean and do I need to make files in this format?

UTF-8 is a way to encode text in a file. There are many different encodings. Basically, a letter like A can be represented by different binary sequences depending on the encoding. This is why when converting some documents have the "s show up as ?. The representation of the " in one encoding is a different character in a different encoding.

UTF-8 is a widely used and lightweight way to handle a large number of characters including unicode characters. It is recommended to use UTF-8 for TXT files. Calibre defaults to utf-8 for TXT files. TXT input will try to auto detect the encoding but due to the way encoding works it's far from perfect. This is also why a conversion solution is often to specify the encoding before hand.

[Agama]

I have found that Notepad++ will allow me to encode in both "UTF-8" and "UTF-8 without BOM". Is one of these encodings preferred over the other?

[user_none]

Originally Posted by Agama

I have found that Notepad++ will allow me to encode in both "UTF-8" and "UTF-8 without BOM". Is one of these encodings preferred over the other?

Without BOM. Typically editors default to utf-8 and have an option to do it with the BOM.