Help with content modifying plugin

[oblib__]

I hope this isn't the wrong place to ask, or to simple/demanding of a question, but I'd like to make a Calibre plugin that can do mass replacements of strings within the text of a book, independent of incoming or outgoing format. I've got the heart of the conversion written in python already, in that if I pass my python script a text file (or even just a string), it does the replacements and prints out or saves the result.

What I'm missing is how to do the plugin. Could someone either point me to a similar plugin that I could copy and learn from, or even better, provide me with the skeleton outline of what I need to do this? The "write your own plugin" page gives me the basics, and I would think that I could take the HelloWorld plugin and make it do what I want with just a few changes.

So here's the heart of HelloWorld. How do I make it provide me with the contents of the book (as in all of the words in the book) as a string, and then let me save a modified string as the new contents?

 def run(self, path_to_ebook):
        from calibre.ebooks.metadata.meta import get_metadata, set_metadata
        file = open(path_to_ebook, 'r+b')
        ext  = os.path.splitext(path_to_ebook)[-1][1:].lower()
        mi = get_metadata(file, ext)
        mi.publisher = 'Hello World'
        set_metadata(file, mi, ext)
        return path_to_ebook

Thanks in advance!

[kovidgoyal]

Why aren't you using the builtin search and replace options in the conversion pipeline?

[oblib__]

Not enough entries through the GUI at least. I didn't try researching command line options.

My script has a lots of replacements (182), including some conditional ones (if I see this text in the book, do this, otherwise do that). Could I do 10s to 100s of replacements using the builtin replace feature?

[kovidgoyal]

you can do as many as you want.

[oblib__]

But I can't do the conditionals, right?

I see the parameter --search-replace for ebook-convert, which I'm guessing you are recommending. Is there some way to add my list of regex's so that it executes when I run a convert through the GUI?
See next post

[oblib__]

Actually, looking at my code (I actually wrote the heart of this over a year ago), I've got custom processes for every replacement. I probably just don't know how to use regex's properly, but I wanted to be able to either preserve case, or sometimes lower case, etc.

For example if I wanted to replace all instances of 'hello' with 'howdy', I want Hello to become Howdy, HELLo to become HOWDy, and HELLO to become HOWDY. However with some replacements I wanted to change only subsequent case, so that Hello became Howdy, and HELLO became Howdy. I couldn't see a way to do that in regular expressions without a processing function.

Bottom line, is that now that I look at my code, I can't easily change it into a list of regular expressions.

[kovidgoyal]

Then dont create a plugin and instead run calibre from source and simply modify the preprocess part (wherethe regexes are applied). Much easier than creatinf a plugin.

[oblib__]

So I've been working on this off and on for a while now, and could use some more help. First off, it would be nice if a new type of plugin were created maybe called mid-processing or something. It would be called in the preprocess class (in preprocess.py) just before the final 'return html' in the __call__ function. The plugin would take in the html text, and return a modified html text. I would try and add it myself and submit my changes, but I'm having trouble running from source right now.

As a work around for now, I'm trying to write my plugin to do all the needed opening, modifying, and saving of the book contents, independent of type input type or output type. What is the easiest way to do that? I've tried grabbing relevant code from the plumber.py file, and can successfully create the oeb in a temp directory, but I decided rather than keep hacking at it once every couple of months, maybe it was time to ask for help.

So bottom line is that I'd like to write a plugin that can take any book format in, modify the contents of that book with my python script, and then return the modified contents for final creation by the rest of calibre processing. Thanks in advance!

Oh, and the reason I don't want to just modify and run from source -- I'd like to be able to do this on all my installations (both Linux and Windows) and keep on doing it though future versions without having to redownload and modify code. Basically I want Calibre to always be able to do this work for me, and if it's useful, share it with others.

[kovidgoyal]

Modifying source code and having your changes preserved through updates is easy. Just use bzr merge to update the source and it will preserve your changes, unless there is a conflicting change (which is unlikely since this part of the calibre code is pretty stable).

There is no shortcut to magically explode to html and repack to arbitrary format. You have to run a full conversion pipeline for that. You can do it for particular formats, see the tweak book function in calibre or the modify epub plugin, but in the general case a full pipeline run is required, as some output plugins depend on the normalization that occurs during a pipeline run.

[oblib__]

That helps -- the tweak file helps get epubs going at least, and I can try converting everything to epub.

What do you think about creating a new text-modifying plugin type?

[kovidgoyal]

In principle, i have no objections to a new plugin type, but, I am not very motivated to implement it as I have other priorities and you;re the only person that has asked for it so far. So unless more people ask, or you submit a patch, it is not likely to happen.