Koreader is poor in handling Internet Archive books

[Frenzie]

Some quick experimentation shows that key and value seem to be reversed compared to the older version.

// tested in MuPDF 1.24.3
if (scriptArgs.length < 2) {
	print("usage: mutool run dejavu.js input.pdf output.pdf");
	quit();
}

var bgPix = new Pixmap(ColorSpace.DeviceGray, [0,0,1,1], false);
var fgPix = new Pixmap(ColorSpace.DeviceGray, [0,0,1,1], false);
bgPix.clear(255);
fgPix.clear(0);

var doc = new PDFDocument(scriptArgs[0]);
var bgImg = doc.addImage(new Image(bgPix));
for (var i = 0; i < doc.countPages(); ++i) {
	var page = doc.findPage(i);
	page.Resources.XObject.forEach(function (xobj, name) {
		var mask = xobj.SMask;
		if (mask) {
			// In newer versions this seems to say "Error: truncated jbig2 segment header"
			// var fgImg = doc.addImage(new Image(fgPix, doc.loadImage(mask)));

			// Quick workaround for the above problem
			var fgImg = doc.graftObject(mask);

			page.Resources.XObject[name] = fgImg;
		} else {
			page.Resources.XObject[name] = bgImg;
		}
	});
}
doc.save(scriptArgs[1], "garbage=compact,compress");

[DanCa]

The mutool version was the issue. 1.23 does not work, but 1.21 did. Thank you very much!


I have added a script that converts all files in a given location. Warning: I made this to be run on my e-reader, so it doesn't keep the originals.

Spoiler:

import os
import shutil
import subprocess
# Delete scanned JPX (JPEG 2000) image layers in archive.org pdfs
# Warning: This script does not keep the original files


# Tested with mupdf 1.21. Does not work with 1.23. 

# path to mutool.exe
mutool = r"C:\mupdf-1.21.0-windows\mutool.exe"

def checkForJPX(filename):
    '''Check if the first page of the file contains a JPX layer.'''
    info_output = subprocess.run([mutool, 'info', filename, '1'], capture_output=True)
    if '[ JPX ]' in str(info_output.stdout):
        return True
    else:
        return False


def convertFile(filename):
    '''Remove scanned image layers from archive.org pdfs'''
    print('Working on: ', filename)
    if checkForJPX(filename):
        print('Trying to convert')
        tmpfile = filename + '_tmp.pdf'
        info_output = subprocess.run([mutool, 'run', 'dejazap.js', filename, tmpfile], capture_output=True)
        if checkForJPX(tmpfile):
            print('ERROR, file still has JPX layer, keeping temp file')
        else:
            shutil.move(tmpfile, filename)
            print('                  ... file converted')
    else:
        print('Does not contain JPX')




# from https://gist.github.com/TheMatt2/faf5ca760c61a267412c46bb977718fa
def walklevel(path, depth = 1):
    """It works just like os.walk, but you can pass it a level parameter
       that indicates how deep the recursion will go.
       If depth is 1, the current directory is listed.
       If depth is 0, nothing is returned.
       If depth is -1 (or less than 0), the full depth is walked.
    """
    # If depth is negative, just walk
    # Not using yield from for python2 compat
    # and copy dirs to keep consistant behavior for depth = -1 and depth = inf
    if depth < 0:
        for root, dirs, files in os.walk(path):
            yield root, dirs[:], files
        return
    elif depth == 0:
        return

    # path.count(os.path.sep) is safe because
    # - On Windows "\\" is never allowed in the name of a file or directory
    # - On UNIX "/" is never allowed in the name of a file or directory
    # - On MacOS a literal "/" is quitely translated to a ":" so it is still
    #   safe to count "/".
    base_depth = path.rstrip(os.path.sep).count(os.path.sep)
    for root, dirs, files in os.walk(path):
        yield root, dirs[:], files
        cur_depth = root.count(os.path.sep)
        if base_depth + depth <= cur_depth:
            del dirs[:]

if __name__=='__main__':
# set inputFolder to your e-reader's location
    inputFolder = r'F:\\' # Convert all pdf documents in this folder unless they don't contain JPX layers. Do not keep originals
    max_depth = 3
    print('Converting files in folder:', inputFolder)
    for root, _, files in walklevel(inputFolder, max_depth):
        for f in files:
            if not f.endswith('.pdf'):
                continue
            convertFile(os.path.join(root, f))

[jonnyl]

@Frenzie: Using your script above I get white text on black background, but besides that, it works great. Is that how it's supposed to come out? Is there any way to invert it? My mutool version is 1.23.10.

Swapping the .clear() numbers didn't change anything.

[Frenzie]

Is there any way to invert it?

Only on 1.21 using the previous script I posted, or if you can figure out a way around the issue I noted in the comments:

			// In newer versions this seems to say "Error: truncated jbig2 segment header"
			// var fgImg = doc.addImage(new Image(fgPix, doc.loadImage(mask)));

			// Quick workaround for the above problem
			var fgImg = doc.graftObject(mask);

I suspect that won't be possible without diving into the MuPDF source though.

[jonnyl]

I got v1.21 for Windows, and it's working beautifully now. Thanks!

[jonnyl]

Unfortunately, the script only works on some PDFs. On others, I only get a whole book's worth of empty white pages. I ran `mutool extract` without the script on both kinds of books but I can't tell a difference. All output 2 blurry pages and 1 monochrome inverted legible image (the only one needed, but inverted back to black text-on-white if possible) per page, and in the same order.

Is there anything I could do to make the script work on the other kind of books as well?

[Frenzie]

You'd have to link some of these other books first. Though my guess would be they don't separate the "text" and the "page background" into two layers but simply have it all as one image.

[jonnyl]

I found the problem. The one not working had the monochrome mask images typed as 'mask', and the working one as 'smask' (according to `pdfimages -list`). I just edited "var mask = xobj.SMask;" to "var mask = xobj.Mask;" and to my astonishment that worked

[Frenzie]

In that case you can probably just write it like:

var mask = xobj.SMask || xobj.Mask;

(or the other way around)

[jonnyl]

Confirmed this works. Thanks!