Re-merge files split by Calibre conversion?

[Siavahda]

Apologies, because this has must have been asked before, but I couldn't find it mentioned anywhere - probably not using the right search terms.

I've been given an EPUB that's gone through Calibre conversion at some point, and many of the files have been split - the chapter number in one file, body of the text in the next.

Is there any way to mass re-merge these files? I've always done it manually, one pair of split files at a time, which is time-consuming and painful, since I have hand damage. Finally occurred to me to ask if there was another way!

Maybe some instruction I can give during another conversion...?

[kovidgoyal]

click edit book, shift click to select all the files you want merged, right click and select merge selected text files.

[Siavahda]

Originally Posted by kovidgoyal

click edit book, shift click to select all the files you want merged, right click and select merge selected text files.

Each chapter has been split in two. If I merge them ALL together, I'll have dozens of chapters in one file. Instead of each chapter being its own file.

I need something that merges each pair back with its pair, not with all the other split files.

Right now, it looks like this

part1_split_1
part1_split_2
part2_split_1
part2_split_2
part3_split_1
part3_split_2

and so on.

If I merge all those together, I'll just have one file called part1_split_1, when what I should have is part1, part2, part3.

(Sorry for over-explaining, I don't know many of the technical terms so I'm never sure if I'm explaining properly.)

[kovidgoyal]

merge them all together and then use splitting tool to automatically re split them based on heading tags or whatever other criteria you need.

[Siavahda]

Originally Posted by kovidgoyal

merge them all together and then use splitting tool to automatically re split them based on heading tags or whatever other criteria you need.

I will try that, thank you!

[Siavahda]

Well, that worked on the files that were initially split, but because the same tags were used elsewhere in the book, now parts that weren't split before, are now split.

Can't overstate how much of a mess this ebook is.

[Sirtel]

Originally Posted by Siavahda

Well, that worked on the files that were initially split, but because the same tags were used elsewhere in the book, now parts that weren't split before, are now split.

Can't overstate how much of a mess this ebook is.

I've had plenty of really messy books from 10-15 years ago. And plenty which I had to split or merge manually, as there was no way to do it automatically with correct results. Plus all the other smart formatting decisions, like the author deciding that the best way to end a chapter was to insert a ton of empty paragraphs after the last sentence. And every chapter had a different number of empty paragraphs, so you couldn't even use mass search & replace to delete them.

Oh, happy days.

[Siavahda]

Originally Posted by Sirtel

I've had plenty of really messy books from 10-15 years ago. And plenty which I had to split or merge manually, as there was no way to do it automatically with correct results. Plus all the other smart formatting decisions, like the author deciding that the best way to end a chapter was to insert a ton of empty paragraphs after the last sentence. And every chapter had a different number of empty paragraphs, so you couldn't even use mass search & replace to delete them.

Oh, happy days.

You, sir, entirely understand the struggle 😂

[PeterT]

I can't help but think you should get in touch with the Author, Publisher or Retailer you obtained the book from and express concerns over the formatting of the material.

[Sirtel]

Originally Posted by PeterT

I can't help but think you should get in touch with the Author, Publisher or Retailer you obtained the book from and express concerns over the formatting of the material.

Depends. It may not be a recently published book; many self-published books were quite messy and poorly formatted 15 years ago, as their authors didn't really have any idea how to format ebooks properly, and several programs currently used for book creating didn't exist yet. It would have been no use to complain about it then, as they obviously didn't know how to do a better job. It was much simpler and faster to fix the books myself.

And if the book was published a decade or more ago, it would be pretty strange to start complaining about it now.

[JSWolf]

I know what the problem is. Whoever did the conversion did not or does not know how to use calibre.

[Sirtel]

Originally Posted by JSWolf

I know what the problem is. Whoever did the conversion did not or does not know how to use calibre.

They presumably converted with default settings, which would split the headings in some books.

[JSWolf]

Originally Posted by Sirtel

They presumably converted with default settings, which would split the headings in some books.

I've done that before way back when. But I've gone back and sorted it properly so that didn't happen.

[thiago.eec]

Originally Posted by Siavahda

Each chapter has been split in two. If I merge them ALL together, I'll have dozens of chapters in one file. Instead of each chapter being its own file.

I need something that merges each pair back with its pair, not with all the other split files.

Right now, it looks like this

part1_split_1
part1_split_2
part2_split_1
part2_split_2
part3_split_1
part3_split_2

and so on.

If I merge all those together, I'll just have one file called part1_split_1, when what I should have is part1, part2, part3.

(Sorry for over-explaining, I don't know many of the technical terms so I'm never sure if I'm explaining properly.)

This can be done using the 'Editor Chains' plugin. You have to create a chain first, but then you can use it with a single click in any other book in the future.

This is the necessary module:

Spoiler:

import re

from qt.core import (QWidget, QVBoxLayout, QGroupBox)

from calibre.ebooks.oeb.base import OEB_DOCS, OEB_STYLES
from calibre.ebooks.oeb.polish.split import merge
from calibre.ebooks.oeb.polish.replace import rename_files
from calibre.gui2.tweak_book import editors

from calibre_plugins.editor_chains.actions.base import EditorAction
from calibre_plugins.editor_chains.scope import scope_names, ScopeWidget, validate_scope, scope_is_headless


def spine_index(container, name):
    idx = 0
    for spine_name, is_linear in container.spine_names:
        idx +=1
        if name == spine_name:
            return idx

class ConfigWidget(QWidget):
    def __init__(self, plugin_action):
        QWidget.__init__(self)
        self.plugin_action = plugin_action
        self._init_controls()

    def _init_controls(self):

        l = self.l = QVBoxLayout()
        self.setLayout(l)

        scope_groupbox = QGroupBox(_('Files to merge'))
        l.addWidget(scope_groupbox)
        scope_l = QVBoxLayout()
        scope_groupbox.setLayout(scope_l)
        self.where_box = ScopeWidget(self, self.plugin_action, orientation='vertical',
                                     headless=self.plugin_action.gui is None,
                                     hide=['selected-text','open','current'])
        scope_l.addWidget(self.where_box)

        l.addStretch(1)
        self.resize(self.sizeHint())

    def load_settings(self, settings):
        if settings:
            self.where_box.where = settings['where']

    def save_settings(self):
        settings = {}
        settings['where'] = self.where_box.where
        return settings

class MergeFiles(EditorAction):

    name = 'Merge Files'
    headless = True

    def run(self, chain, settings, *args, **kwargs):
        names = scope_names(chain, settings['where'])
        print(f'DEBUG: Merging Files\n       names: {names}')
        if len(names) < 2:
            print('Error: You need to select at least 2 files to merge')
            return
        container = chain.current_container
        spine_names = [name for name, is_linear in container.spine_names]
        if names[0] in set(container.manifest_items_of_type(OEB_DOCS)):
            category = 'text'
        elif names[0] in set(container.manifest_items_of_type(OEB_STYLES)):
            category = 'styles'
        else:
            print('Error: Can only merge files of categories: text or styles')
            return
        if category == 'text':
            if not set(names).issubset(set(container.manifest_items_of_type(OEB_DOCS))):
                print('Error: some files are your trying to merge are not of category text')
                return
            if not set(names).issubset(set(spine_names)):
                print('Error: Some files you are trying to merge not in spine')
                return
            names = sorted(names, key=lambda name: spine_index(container, name))
        else:
            if not set(names).issubset(set(container.manifest_items_of_type(OEB_STYLES))):
                print('Error: some files are your trying to merge are not of category styles')
                return
            names = sorted(names)
        master = names[0]
        print(f'DEBUG: Merging Files\n       master: {master}')

        # Create groups with the same prefix before merging
        while names:
            master = names[0]
            group = []
            for name in names[:]:
                master_prefix =  re.search('.*?_split_[^d+]', master)[0]
                if re.search('.*?_split_[^d+]', name)[0] == master_prefix:
                    group.append(name)
                    names.remove(name)
            merge(container, category, group, master)
            new_name = re.sub('_split_.*?\\d+', '', master)
            rename_files(container, {master: new_name})
            #if new_name in editors:
            #    chain.boss.show_editor(new_name)

    def validate(self, settings):
        scope_ok = validate_scope(settings['where'])
        if scope_ok is not True:
            return scope_ok
        return True

    def config_widget(self):
        return ConfigWidget

    def is_headless(self, settings):
        return scope_is_headless(settings['where'])

Instructions to configure the chain:
1) Manage Modules > Create > Choose a name for the module (merge_files) > Clear the contents of the code area > Copy/paste the code above > Ok
2) Add/Modify Chains > Plus sign > Give it a title (Merge splited)
3) Gear icon > Choose an icon (optional) > Plus sign > Double click the 'Action' cell > User Actions > Merge Files
4) Gear icon > Choose 'Filenames' > Include filters > Use this regex:

.*?_split_

5) Click OK in all windows

Now you have a 'Merge splited' option in the Editor Chain menu. You can also set a keyboard shortcut via: Add/Modify Chains > Keyboard shortcuts.