[Editor Plugin] Editor Chains

[nqk]

This is a great Christmas gift for me. Thank you very much.

[Capricorn]

I have been using some simple editor chains for some time now. Some search&replace items. That works well, and saves me time (thanks). But I would like to add some actions that only work on the opf file eg remove page-map. Is this possible or not? My knowledge of editor chains is still in its infancy. And I did not see any posts that clearly indicate this possibility.

[thiago.eec]

Originally Posted by Capricorn

I have been using some simple editor chains for some time now. Some search&replace items. That works well, and saves me time (thanks). But I would like to add some actions that only work on the opf file eg remove page-map. Is this possible or not? My knowledge of editor chains is still in its infancy. And I did not see any posts that clearly indicate this possibility.

Yes, you can achieve this using filename filters. See the example below: it searches only in the OPF and removes the <tours> element.

[Capricorn]

Oh man....that is so cool. I did not know this fuctionality was there.
Thank you.

[Capricorn]

Thiago, In Post #99 you provide code to solve invaild IDs. That is something I would like to automate too, but how does this code work? I have no clue as to the steps needed to do so? I have read your post and the ones before and after, but I am still clueless. My knowledge is definitely not up to scratch. Any chance as to a step by step explanation how to do this? Would be much appreciated. Thanks in advance.

[thiago.eec]

Originally Posted by Capricorn

Thiago, In Post #99 you provide code to solve invaild IDs. That is something I would like to automate too, but how does this code work? I have no clue as to the steps needed to do so? I have read your post and the ones before and after, but I am still clueless. My knowledge is definitely not up to scratch. Any chance as to a step by step explanation how to do this? Would be much appreciated. Thanks in advance.

First you need to create the Module via Manage Modules > Create > Replace the existing text by this:

Spoiler:

from qt.core import (Qt, QApplication)

from calibre.ebooks.oeb.polish.check import parsing
from calibre.ebooks.oeb.polish.check.main import fix_errors

from calibre_plugins.editor_chains.actions.base import EditorAction

class ReplaceIDs(EditorAction):

    name = 'Replace IDs'
    headless = True

    def run(self, chain, settings, *args, **kwargs):
        container = chain.current_container
        QApplication.setOverrideCursor(Qt.WaitCursor)
        QApplication.processEvents()
        errors = parsing.check_ids(container)
        changed = fix_errors(container, errors)
        QApplication.restoreOverrideCursor()

Then, you need to create the chain, via Add/Modify Chains > Plus sign (+) > Choose a name > Gear icon (Settings) > You can choose an icon or not > Plus sign (+) > Select the new line > Double click in the Action field > User actions > Replace IDs > Ok.

Now, the new action will appear as option in the main menu. You can also use it as part of more complex chain, like the one you asked about before, doing search and replace.

[Capricorn]

A very big thank you for posting this.
So the "Manage Modules" actually allows to add new code to the plugin. I have always wondered why it was there.
It is now working for me. One item less to manually do.

[Capricorn]

Thiago, In post 99 you have several zip files attached. One of them is "clean up.zip". On looking at it, I see a json file that contains interesting stuff I would like to automate too. I tried to add it to my Manage Modules in editor chains but I get the error message some stuff is missing. Hence I assume it is not code for a module in editor chains. But the question is then: how do I add this to the editor chain? Or are these all search & replace statements? And how would I add it to my saved search & replace statements? Do I need to merge it with some calibre json file?

[thiago.eec]

Originally Posted by Capricorn

Thiago, In post 99 you have several zip files attached. One of them is "clean up.zip". On looking at it, I see a json file that contains interesting stuff I would like to automate too. I tried to add it to my Manage Modules in editor chains but I get the error message some stuff is missing. Hence I assume it is not code for a module in editor chains. But the question is then: how do I add this to the editor chain? Or are these all search & replace statements? And how would I add it to my saved search & replace statements? Do I need to merge it with some calibre json file?

There are three different kind of code posted there:

1) Modules: they were posted inside the spoiler tag; click 'show' to view the code; add them to your modules. Some of those are used in the 'Cleanup' chain, so if not present the chain will fail. Modules are not chains, so they can't be used directly. You neeed to create a chain and configure it to use the respective module (like the 'Replace IDs' chain).

2) Regex function: there is only one posted. This is used to find <img> tags missing the alt attribute. To use it in the chain, add this function to the editor first: open the search tool (CTRL+F) and change mode to Regex Function; then click on 'Create/edit'; replace the content with the provided code; give this function a name (it will be used in the 'Cleanup' chain).

3) ZIP files: those are the chain files. You can import then directly via Add/modify Chains > Right click in the table > Import. After you add the 'Cleanup' chain, click in the gear icon. You can remove actions you don't need and add your own search/replace actions. The missing 'alt' action needs to be configured, pointing to your Regex function created in step 2.

I hope this covers everything.

[dewd]

When running using Action Chains, this (optionally) creates a backup. What is the best way to restore this backup if needed?

[slimcat]

Originally Posted by dewd

When running using Action Chains, this (optionally) creates a backup. What is the best way to restore this backup if needed?

I just had to restore ~1000 epubs after Calibre crashed in the middle of an Action Chain run on my whole library, so I just had to figure this out. The EC_BACKUP is just a copy of the original EPUB saved with a different format name (if you save the backup to disk and then change the ec_backup file extension back to epub you'll have a working epub copy of your backup).

Not sure if there's a better/easier way to restore the backup within Calibre, but the way I ended up doing it was with an Action Chain:

Action: Save/Restore Formats
Settings:
- Source format: ec_backup
- Destination format: epub

Check "Confirm overwriting existing format confirmation dialog" if you want Calibre to ask before replacing any existing EPUBs, otherwise they're silently overwritten. If you want the backup to be automatically deleted after the restore, check "Remove source format after copying."

I've attached the Action Chain I used. It made it super easy to restore everything.

On a related note, something I only caught shortly before this whole thing happened: if you're running a chain with multiple Editor Chain actions in it and want a backup copy of the EPUB before anything is changed, only check the backup option on the first Editor Chain. Backup is selected by default on each Editor Chain action, so you have to remember to uncheck it on the steps after the 1st one.

[Capricorn]

Originally Posted by thiago.eec

There are three different kind of code posted there:

1) Modules: they were posted inside the spoiler tag; click 'show' to view the code; add them to your modules. Some of those are used in the 'Cleanup' chain, so if not present the chain will fail. Modules are not chains, so they can't be used directly. You neeed to create a chain and configure it to use the respective module (like the 'Replace IDs' chain).

2) Regex function: there is only one posted. This is used to find <img> tags missing the alt attribute. To use it in the chain, add this function to the editor first: open the search tool (CTRL+F) and change mode to Regex Function; then click on 'Create/edit'; replace the content with the provided code; give this function a name (it will be used in the 'Cleanup' chain).

3) ZIP files: those are the chain files. You can import then directly via Add/modify Chains > Right click in the table > Import. After you add the 'Cleanup' chain, click in the gear icon. You can remove actions you don't need and add your own search/replace actions. The missing 'alt' action needs to be configured, pointing to your Regex function created in step 2.

I hope this covers everything.

Ah..that clears up a lot of confusion in my brain. And I did not even know I could import those chain files.

[natruok]

Hi! Thank you so much for the plugin — I was wondering if there's a way to add a new file at the top/beginning of the epub? I've been testing all the options I can try, but the only way to add a file at the beginning appears to be through the "Insert Inline TOC" option, but that doesn't quite do the same as adding a file from a template. I tried searching a few keywords in this thread but doesn't seem like the question has come up before.

[thiago.eec]

Originally Posted by natruok

Hi! Thank you so much for the plugin — I was wondering if there's a way to add a new file at the top/beginning of the epub? I've been testing all the options I can try, but the only way to add a file at the beginning appears to be through the "Insert Inline TOC" option, but that doesn't quite do the same as adding a file from a template. I tried searching a few keywords in this thread but doesn't seem like the question has come up before.

I've created an action that uses a modified version of the plugin's 'Add Files' action. It has two differences from the original version:

1. Files are added respecting the existing folder structure (Text, Fonts, Styles, etc.)
2. If you are adding a text file, you can choose where to put it: at the beginning, at the end, after the currently editing file, or at a custom position (0 - 9). This only works for single file mode.

To use it, first add the module below, then create the chain. You'll see a new action called 'Add Files 2' under 'User Actions'.

Module:

Spoiler:

import copy
import os
import regex

from qt.core import (QApplication, Qt, QWidget, QHBoxLayout, QVBoxLayout, QGridLayout,
                     QGroupBox, QToolButton, QCheckBox, QLineEdit, QComboBox, QLabel,
                     QRadioButton)

from calibre import prints
from calibre.constants import DEBUG, iswindows
from calibre.gui2 import error_dialog, choose_files, choose_dir
from calibre.gui2.tweak_book.boss import get_boss

from calibre_plugins.editor_chains.actions.base import EditorAction
from calibre_plugins.editor_chains.common_utils import (get_file_path,
    DragDropComboBox, get_icon)
from calibre.ebooks.oeb.base import OEB_DOCS, OEB_STYLES
from calibre.ebooks.oeb.polish.replace import get_recommended_folders
from calibre.ebooks.oeb.polish.utils import link_stylesheets, guess_type

try:
    load_translations()
except NameError:
    prints("EditorChains::actions/add_files.py "
           "- exception when loading translations")


def add_file_to_container(container, filename, target_name=None,
                          if_name_exists='skip', link_css=False,
                          link_html_to_css=False, spine_position=None):

    file_path = get_file_path(filename)
    ext = os.path.splitext(file_path)[-1].lstrip('.').lower()
    if not target_name:
        target_name = os.path.basename(file_path)

    # Use the book folder structure accordingly
    fmap = get_recommended_folders(container, (target_name,))
    if fmap[target_name]:
        target_name = '/'.join((fmap[target_name], target_name))

    modify_name_if_needed = False
    if container.has_name(target_name):
        if if_name_exists == 'skip':
            return
        elif if_name_exists == 'rename':
            modify_name_if_needed = True
        elif if_name_exists == 'overwrite':
            container.remove_item(target_name)

    spine_index = None  # If a None value is used, the file will be put at the end
    if spine_position:
        if spine_position == 'beginning':
            spine_index = 0
        elif spine_position == 'end':
            spine_index = len(container.manifest_items) - 1
        elif spine_position == 'after_current':
            boss = get_boss()
            spine_index = container.index_in_spine(boss.currently_editing or '')
            if spine_index is not None:
                spine_index += 1
        else:
            spine_index = int(spine_position)

    with open(file_path, 'rb') as f:
        data = f.read()
        added_name = container.add_file(target_name, data, spine_index=spine_index,
                                        modify_name_if_needed=modify_name_if_needed)

        if ext == 'css' and link_css:
            all_text = list(container.manifest_items_of_type(OEB_DOCS))
            for x in all_text:
                for y in ['nav.xhtml','titlepage.xhtml']:
                    if x.endswith(y):
                        all_text.remove(x)
            link_stylesheets(container, all_text, [added_name])
            for x in all_text:
                container.dirty(x)
        elif ext in ['html', 'xhtml'] and link_html_to_css:
            sheets = list(container.manifest_items_of_type(OEB_STYLES))
            link_stylesheets(container, [added_name], sheets)
            container.dirty(added_name)


class ConfigWidget(QWidget):
    def __init__(self, plugin_action):
        QWidget.__init__(self)
        self.plugin_action = plugin_action
        self.gui = plugin_action.gui
        self.mode = 'file'
        self._init_controls()

    def _init_controls(self):

        l = self.l = QVBoxLayout()
        self.setLayout(l)

        self.file_box = QGroupBox(_('Choose file(s):'))
        l.addWidget(self.file_box)
        self.file_box_layout = QVBoxLayout()
        self.file_box.setLayout(self.file_box_layout)
        hl1 = QHBoxLayout()
        self.file_box_layout.addLayout(hl1)
        self.file_opt = QRadioButton(_('Select a file to add'))
        hl1.addWidget(self.file_opt)
        self.dir_opt = QRadioButton(_('Select a folder to add'))
        hl1.addWidget(self.dir_opt)
        self.file_opt.setChecked(True)
        for x in [self.file_opt, self.dir_opt]:
            x.toggled.connect(self._mode_toggled)
        file_layout = self.file_layout = QHBoxLayout()
        self.file_box_layout.addLayout(file_layout)
        self.file_combo = DragDropComboBox(self, drop_mode='file')
        file_layout.addWidget(self.file_combo)
        self.choose_file_button = QToolButton(self)
        self.choose_file_button.setToolTip(_('Choose file'))
        self.choose_file_button.setIcon(get_icon('document_open.png'))
        self.choose_file_button.clicked.connect(self._choose_file)
        file_layout.addWidget(self.choose_file_button)

        self.name_groupbox = QGroupBox(_(
            'Filename and path inside ebook (optional)'))
        self.name_groupbox.setToolTip(_(
            'Determine the filename and path inside the ebook. '
            'e.g. Images/image-1.png.\n'
            'If not set, filename will not be modified '
            'and the target file will reside in the root of the container'))
        name_l = QVBoxLayout()
        self.name_groupbox.setLayout(name_l)
        self.name_groupbox.setCheckable(True)
        self.name_groupbox.setChecked(False)
        l.addWidget(self.name_groupbox)
        self.target_exists_le = QLineEdit()
        name_l.addWidget(self.target_exists_le)

        self.exists_groupbox = QGroupBox(_(
            'What to do if same filename exists in ebook.'))
        exists_l = QVBoxLayout()
        self.exists_groupbox.setLayout(exists_l)
        l.addWidget(self.exists_groupbox)
        self.skip_opt = QRadioButton(_(
            'Don\'t add the file'))
        self.skip_opt.setChecked(True)
        exists_l.addWidget(self.skip_opt)
        self.overwrite_opt = QRadioButton(_(
            'Overwrite existing file'))
        exists_l.addWidget(self.overwrite_opt)
        self.modify_opt = QRadioButton(_(
            'Add and modify name'))
        exists_l.addWidget(self.modify_opt)

        self.css_groupbox = QGroupBox(_('CSS Options'))
        css_l = QVBoxLayout()
        self.css_groupbox.setLayout(css_l)
        l.addWidget(self.css_groupbox)

        self.css_link_chk = QCheckBox(_(
            'Link added css file to html files in book.'))
        self.css_link_chk.setChecked(False)
        css_l.addWidget(self.css_link_chk)

        self.html_link_chk = QCheckBox(_(
            'Link added XHTML file to css sheets already in container'))
        self.html_link_chk.setChecked(False)
        css_l.addWidget(self.html_link_chk)

        # Spine position
        self.spine_groupbox = QGroupBox(_(
            'Spine position for text files.'))
        spine_l = QGridLayout()
        self.spine_groupbox.setLayout(spine_l)
        l.addWidget(self.spine_groupbox)
        self.beginning_opt = QRadioButton(_(
            'At the beginning'))
        spine_l.addWidget(self.beginning_opt, 0, 0)
        self.end_opt = QRadioButton(_(
            'At the end'))
        spine_l.addWidget(self.end_opt, 1, 0)
        self.after_current_opt = QRadioButton(_(
            'After the currently editing text file'))
        self.after_current_opt.setToolTip(_('If there is no text file tab active, the new file goes to end'))
        self.after_current_opt.setChecked(True)
        spine_l.addWidget(self.after_current_opt, 2, 0)
        self.custom_opt = QRadioButton(_('Custom position'))
        tooltip = _('First position is 0. Last position will be used if chosen value is bigger then spine length.')
        self.pos_combobox = QComboBox()
        self.pos_combobox.setToolTip(tooltip)
        self.custom_opt.setToolTip(tooltip)
        self.pos_combobox.addItems({'0', '1', '2', '3', '4', '5', '6', '7', '8', '9'})
        self.pos_combobox.model().sort(0)
        self.pos_combobox.setCurrentText('1')
        self.pos_combobox.setVisible(False)
        for x in [self.beginning_opt, self.end_opt, self.after_current_opt, self.custom_opt]:
            x.toggled.connect(self._position_toggled)
        spine_l.addWidget(self.custom_opt, 3, 0)
        spine_l.addWidget(self.pos_combobox, 3, 1)

        l.addStretch(1)

        self.setMinimumSize(350, 670)

    def _position_toggled(self):
        self.pos_combobox.setVisible(self.custom_opt.isChecked())

    def _mode_toggled(self):
        if self.file_opt.isChecked():
            self.mode = 'file'
        else:
            self.mode = 'dir'
        self.name_groupbox.setVisible(self.file_opt.isChecked())
        self.spine_groupbox.setVisible(self.file_opt.isChecked())
        self.file_combo.setCurrentText('')

    def _choose_file(self):
        if self.mode == 'file':
            chosen_files = choose_files(
                None, _('Select file dialog'), _('Select a file'),
                all_files=True, select_only_single_file=True)
            if not chosen_files:
                return

            file_path = chosen_files[0]

        else:
            file_path = choose_dir(self, 'choose directory',
                    _('Choose directory'))
            if not file_path:
                return

        if iswindows:
            file_path = os.path.normpath(file_path)

        self.block_events = True
        existing_index = self.file_combo.findText(file_path, Qt.MatchExactly)
        if existing_index >= 0:
            self.file_combo.setCurrentIndex(existing_index)
        else:
            self.file_combo.insertItem(0, file_path)
            self.file_combo.setCurrentIndex(0)
        self.block_events = False

    def load_settings(self, settings):
        if settings:
            is_file = settings.get('is_file', True)
            self.file_opt.setChecked(is_file)
            self.dir_opt.setChecked(not is_file)
            filename = settings.get('filename')
            if filename:
                self.file_combo.setCurrentText(filename)
            if_name_exists = settings.get('if_name_exists')
            if if_name_exists == 'skip':
                self.skip_opt.setChecked(True)
            elif if_name_exists == 'overwrite':
                self.overwrite_opt.setChecked(True)
            elif if_name_exists == 'rename':
                self.modify_opt.setChecked(True)
            self.css_link_chk.setChecked(settings['link_css'])
            self.html_link_chk.setChecked(
                settings.get('link_html_to_css', False))
            spine_position = settings.get('spine_position')
            if spine_position == 'beginning':
                self.beginning_opt.setChecked(True)
            elif spine_position == 'end':
                self.end_opt.setChecked(True)
            elif spine_position == 'after_current':
                self.after_current_opt.setChecked(True)
            else:
                self.custom_opt.setChecked(True)
                self.pos_combobox.setCurrentText(settings.get('spine_position'))

            if is_file:
                target_name = settings.get('target_name', '')
                if target_name:
                    self.name_groupbox.setChecked(True)
                    self.target_exists_le.setText(target_name)

    def save_settings(self):
        settings = {}
        settings['is_file'] = self.file_opt.isChecked()
        settings['filename'] = self.file_combo.currentText().strip()
        if self.skip_opt.isChecked():
            settings['if_name_exists'] = 'skip'
        elif self.overwrite_opt.isChecked():
            settings['if_name_exists'] = 'overwrite'
        elif self.modify_opt.isChecked():
            settings['if_name_exists'] = 'rename'
        settings['link_css'] = self.css_link_chk.isChecked()
        settings['link_html_to_css'] = self.html_link_chk.isChecked()
        if self.beginning_opt.isChecked():
            settings['spine_position'] = 'beginning'
        elif self.end_opt.isChecked():
            settings['spine_position'] = 'end'
        elif self.after_current_opt.isChecked():
            settings['spine_position'] = 'after_current'
        else:
            settings['spine_position'] = self.pos_combobox.currentText()
        if self.file_opt.isChecked() and self.name_groupbox.isChecked():
            settings['target_name'] = self.target_exists_le.text()
        return settings


def tree_names(tree, include_dirs=False):
    l = []
    for parent, dirs, files in os.walk(tree):
        iter_over = files
        if include_dirs: iter_over += dirs
        for x in iter_over:
            full_path = os.path.join(parent, x)
            l.append(full_path)
    return l


class AddFiles(EditorAction):

    name = 'Add Files 2'
    headless = True

    def run(self, chain, settings, *args, **kwargs):
        container = chain.current_container

        is_file = settings.get('is_file', True)
        filename = settings.get('filename')
        if_name_exists = settings.get('if_name_exists', 'skip')
        link_css = settings.get('link_css', False)
        link_html_to_css = settings.get('link_html_to_css', False)
        spine_position = settings.get('spine_position', 'after_current')
        target_name = settings.get('target_name')

        if is_file:
            add_file_to_container(
                container,
                filename,
                target_name=target_name,
                if_name_exists=if_name_exists,
                link_css=link_css,
                link_html_to_css=link_html_to_css,
                spine_position=spine_position
            )
        else:
            tree = filename
            for fpath in sorted(tree_names(tree)):
                relpath = os.path.relpath(fpath, tree)
                add_file_to_container(
                    container,
                    fpath,
                    target_name=relpath,
                    if_name_exists=if_name_exists,
                    link_css=link_css,
                    link_html_to_css=link_html_to_css,
                    spine_position = spine_position
                )

    def validate(self, settings):
        if not settings:
            return (
                _('Settings Error'),
                _('You must configure this action before running it')
            )
        filename = settings.get('filename')
        if filename:
            file_path = get_file_path(filename)
            if not os.access(file_path, os.R_OK):
                return (_
                        ('Invalid path'),
                        _(f'Path: {file_path} is not a valid path')
                )
        else:
            return (
                _('Missing file/folder'),
                _('You must choose a valid file/folder')
            )
        if settings.get('is_file', True):
            if not os.path.isfile(file_path):
                return (
                    _('File error'),
                    _(f'Path: {filename} does not belong to a file')
                )
            target_name = settings.get('target_name')
            if not target_name:
                target_name = os.path.basename(file_path)
            targets = [target_name]
        else:
            if not os.path.isdir(file_path):
                return (
                    _('Directory error'),
                    _(f'Path: {filename} does not belong to a directory')
                )
            targets = [os.path.relpath(x, file_path) for x in
                       tree_names(file_path)]
            if not targets:
                return (
                    _('Empty directory'),
                    _(f'Directory: {filename} does not contain any file')
                )
        for target_name in targets:
            if '..' in target_name:
                return (
                    _('Invalid target name'),
                    _(f'Target name ({target_name}) '
                      'contains .. which is not allowed')
                )
            for x in target_name.split('/'):
                if not regex.search(r'^[\w\d\-\._]+$', x):
                    return (
                        _('Invalid target name'),
                        _(f'Target name ({target_name}) contains '
                          'illegal characters')
                    )
        return True

    def config_widget(self):
        return ConfigWidget