[GUI Plugin] EpubMerge

[PeterT]

By failed, what exactly happened?

[JimmXinu]

Originally Posted by randy@asa3.org

I did the debug process but the log indicated nothing wrong but still output failed. I used exactly the same files on my own laptop which was running Calibre 7.6 instead of 7.16 and EpubMerge v2.17 instead of 3.1. It worked fine.

But on the desktop with versions 7.16 and 3.1, the same inputs and settings leads to "EpubMerge failed. No new Epub file was created..."

The debug log should have given you something.

If it didn't, then I can't really help you without seeing the input epubs.

If you don't want to share the epubs due to copyright issues, you could try applying the ScrambleEbook plugin and merging the scrambled versions. If they fail the same way, you can share the scrambled versions for me to see.

[randy@asa3.org]

By failed, I mean when I activate the EpubMerge plugin, put in all the metadata and click ok, instead of getting the usual response of completion, I get a banner that says "EpubMerge failed. No new Epub file was created..." but without any error message or indication of why it failed. In running the plugin in debug mode, after quitting Calibre, a log file was created but it gave no clue that there was something wrong. Is there a way I can post that log file?

Is there a way I can revert to version 2.17 of the plugin to try that instead of 3.1? And/or perhaps Calibre 7.6 instead of 7.16? Those versions worked fine with my Windows 11 laptop but did not work on the two Windows 10 desktops at the office that I support.

[theducks]

EPUB Merge seems to run in background (Worker process?)
I just tested a 3 book merge on 3.1 (w10)

I filled in the metadata.
Then I got a message that it would complete in background BEFORE it would appear (as a format) in the library.
Then I got the confirmation, and it was there

did you start another process before completion (the confirmations might have submerged, not allowing confirmation)
If something was blocking 'workers', I would expect other things would also fail.

[JimmXinu]

MR doesn't allow very new users to post files. There's a Requests to enable Private Messages thread. Or use one of the many file hosting solutions available online and post a link here.

Yes, previous versions of both are available:

https://download.calibre-ebook.com/

https://github.com/JimmXinu/EpubMerge/releases

[randy@asa3.org]

The 7 Epub files we need to merge as well as the text basis for the metadata file and the cover page we use are all in this folder:
https://www.asa3.org/ASA/Epub/

I did download the v2.17 of the plugin but I don't know how to install it in Calibre. How do I do that? I presume I remove v3.1 but how do I get it to install the older version? I'm just trying to see if the version makes a difference. The desktops have merged files exactly like the ones in the above folder for about 5 years. But after updating to the latest versions, we get the failure response I mentioned.b

Randy

[randy@asa3.org]

PS I just uploaded a photo of the failure notice I get to the folder in the link in m previous post. This is the only indication, other than a blank file added, that there was a failure. The file name is /ASA/Epub/EpubMerge Failure notice.pdf

[randy@asa3.org]

And I added the log file of the debug mode: file name is tmpsbo5__ug.rtf

[JimmXinu]

First problem I found: PSCF9-24p086-106Helble.epub has a bad CRC. Log error:

Spoiler:

EpubMerge: PSCF9-24p086-106Helble Anthology
        EpubMerge: DEBUG: 2024-08-21 19:32:00,782: calibre_plugins.epubmerge.epubmerge.epubmerge(273): prep output:0.011002302169799805
        EpubMerge: DEBUG: 2024-08-21 19:32:00,786: calibre_plugins.epubmerge.epubmerge.epubmerge(451): write rootfile OEBPS/content.opf to 1/OEBPS/content.opf
        EpubMerge: DEBUG: 2024-08-21 19:32:00,790: calibre_plugins.epubmerge.epubmerge.epubmerge(488): write toc.ncx OEBPS/toc.ncx to 1/OEBPS/toc.ncx
        Traceback (most recent call last):
          File "runpy.py", line 198, in _run_module_as_main
          File "runpy.py", line 88, in _run_code
          File "site.py", line 83, in <module>
          File "site.py", line 78, in main
          File "site.py", line 50, in run_entry_point
          File "calibre\utils\ipc\worker.py", line 215, in main
          File "calibre\utils\ipc\worker.py", line 150, in arbitrary_n
          File "calibre_plugins.epubmerge.jobs", line 84, in do_merge_bg
          File "calibre_plugins.epubmerge.epubmerge.epubmerge", line 499, in doMerge
          File "zipfile.py", line 1511, in read
          File "zipfile.py", line 941, in read
          File "zipfile.py", line 1045, in _read1
          File "zipfile.py", line 973, in _update_crc
        zipfile.BadZipFile: Bad CRC-32 for file 'OEBPS/image/Figure 9 - Raindrop imp_fmt.png'
        ==================================================

        Beginning Merge...

        ==================================================

        Error occurred processing 'PSCF9-24p086-106Helble' by ['Unknown'].

Fixing it:

This one I fixed just by opening it and saving in Calibre's Edit book.

Second problem: Found this in all of the epubs; the embedded fonts are not encrypted correctly. Log:

Spoiler:

EpubMerge: PSCF9-24p086-106Helble Anthology
        EpubMerge: DEBUG: 2024-08-21 19:34:57,167: calibre_plugins.epubmerge.epubmerge.epubmerge(273): prep output:0.013002634048461914
        EpubMerge: DEBUG: 2024-08-21 19:34:57,170: calibre_plugins.epubmerge.epubmerge.epubmerge(451): write rootfile OEBPS/content.opf to 1/OEBPS/content.opf
        EpubMerge: DEBUG: 2024-08-21 19:34:57,175: calibre_plugins.epubmerge.epubmerge.epubmerge(488): write toc.ncx OEBPS/toc.ncx to 1/OEBPS/toc.ncx
        EpubMerge: INFO: 2024-08-21 19:34:57,450: calibre_plugins.epubmerge.epubmerge.epubmerge(501): Decrypting font file: font/Arial-BoldMT.ttf
        Traceback (most recent call last):
          File "runpy.py", line 198, in _run_module_as_main
          File "runpy.py", line 88, in _run_code
          File "site.py", line 83, in <module>
          File "site.py", line 78, in main
          File "site.py", line 50, in run_entry_point
          File "calibre\utils\ipc\worker.py", line 215, in main
          File "calibre\utils\ipc\worker.py", line 150, in arbitrary_n
          File "calibre_plugins.epubmerge.jobs", line 84, in do_merge_bg
          File "calibre_plugins.epubmerge.epubmerge.epubmerge", line 502, in doMerge
          File "calibre_plugins.epubmerge.epubmerge.epubmerge", line 161, in get_decrypted_font_data
          File "calibre_plugins.epubmerge.epubmerge.epubmerge", line 141, in get_idpf_key
          File "re\__init__.py", line 185, in sub
        TypeError: expected string or bytes-like object, got 'NoneType'
        ==================================================

        Beginning Merge...

        ==================================================

        Error occurred processing 'PSCF9-24p086-106Helble' by ['Unknown'].

(FYI, these messages were available in the background Jobs log and the debug log.)

This isn't just a problem with EpubMerge, Calibre's Edit book and Convert cannot decrypt the fonts in these epubs either. Which isn't too surprising, I lifted the decrypt code for EpubMerge from Calibre.

It's easy to see the corrupted font files by opening the epub in Calibre's Edit book and looking for the 'X on red' icon on each font file in the file browser.

In past, EpubMerge didn't handle encrypted fonts at all--the older version in fact broke all encrypted fonts by changing the uuid used as the encryption key.

Fixing it:

Ideally, you'd fix the embedded fonts in the input epubs. I can't help you with that.

If that's not an option, the quickest solution I found was to remove the embedded fonts using the Modify ePub 'Remove embedded fonts' option (second from top on the right column of options). Then the current EpubMerge works.

Or, of course, you could use the older version of EpubMerge and know you're sending it with broken fonts. The plugin intro thread documents how to manually install plugins.

[PeterT]

Originally Posted by randy@asa3.org

I did download the v2.17 of the plugin but I don't know how to install it in Calibre. How do I do that?

Go to Preferences | Plugins and select the option Load plugin from file and point to the zip you downloaded.

[PeterT]

Originally Posted by randy@asa3.org

PS I just uploaded a photo of the failure notice I get to the folder in the link in m previous post. This is the only indication, other than a blank file added, that there was a failure. The file name is /ASA/Epub/EpubMerge Failure notice.pdf

Did you do as the message says and go to the Calibre Jobs menu and look at the detailed output from the job?

[DNSB]

I took a quick look at the files involved. All 7 of them had the fonts encrypted using the IDPF method. I used Sigil to mark them as unecrypted. In PSCF9-24p086-106Helble.epub, the Figure 9 - Raindrop imp_fmt.png and Helble fig10 top p13_fmt.png 10 images showed errors. I just re-added them to the epub but they should be replaced as they are definitely corrupted.

I've attached the 7 epubs and the anthology I created from them in numerical order. One note is that many of the font files are duplicated through the epubs and I would recommend combining the font files after the merge.

[randy@asa3.org]

Thank you all so very much. It will take me a little while to absorb all of this. I'll work through it and keep in touch. I should add that the component epub files were all generated from InDesign. I'll have to learn more about how they handle the fonts.

[randy@asa3.org]

Thank you all so very much for helping me. I'm learning a lot. Here's an update of what I've done so far:

Removing the embedded fonts on each input file worked in the sense of getting a valid anthology but the output is not acceptable for readability. Those font changes like italics, etc. are necessary.

I have not used InDesign and the users I know do not know how or if the encryption of embedded fonts can be modified.

I then tried various combinations of versions on a desktop running Windows 10 and obtained the following results, always using only the files I put in the folder in the link I posted earlier:

Calibre v7.16 EpubMerge v3.1 Failed
Calibre v7.16 EpubMerge v2.17 Failed
Calibre v7.6 EpubMerge v3.1 Failed
Calibre v7.6 EpubMerge v2.17 SUCCESS!

So in my role as a volunteer IT support for a small non-profit office, I can keep them working by installing C v7.6 EM v2.17 on their desktops and telling them not to upgrade either one until I tell them it's ok.

[JimmXinu]

New Test Version Attached

2024-08-23
- Don't require <dc:identifier> to have attribute opf:scheme="uuid" when using IDPF font encryption.

I still don't know exactly what's going on here--I suspect that there's something weird with the embedded fonts.

I still needed to address the PSCF9-24p086-106Helble.epub bad CRC issue first, but skipping the <dc:identifier opf:scheme="uuid"...> check lets these epubs merge and at least one of the embedded fonts still works--I didn't bother checking more than one.

Calibre's Edit book still marks all the fonts as 'not valid', including the one I can see working.

EpubMerge v2.17 is before EM learned about encrypted fonts, so that makes sense. I've no idea why Calibre 7.6 would make any difference.

Using Sigil to specifically IDPF encrypt embedded fonts in a different epub works fine with Edit book and EpubMerge.