[GUI Plugin] TextDiff

[feuille]

[GUI Plugin] TextDiff - Version 1.2.4 - 01-07-2024

A Calibre GUI plugin for finding text differences in two book formats.

Main features:
--------------
This plugin shows the differences between two selected book formatss.
The formats are first converted to text format (even if the source format is already text) with Calibre's convert utility (https://manual.calibre-ebook.com/gen...k-convert.html).
If the conversion fails, the format has no text content (as scanned PDF files) or Calibre cannot find an appropriate conversion tool (as Microsoft wordconv).
Then the text files obtained this way are read into memory and possibly edited (removing blank lines and other changes as described under "Planned Features".
Then the compare is done with Python's DiffLib (https://docs.python.org/3/library/difflib.html).
The ratio gives a measure for the similarity of the two texts. 1.0 means the texts are identical, A value near 0.0 means, that the texts are complete different.
The last thing may also occur, when the source format has no text content (as scanned PDF files). Then one should create a new book format (text) with an OCR process.

The detailed workflow is as follows:
1. Select a book with at least two formats or two books with at least one format each to compare.
2. Chose two formats.
3. Chose the output format and other comparison options.
4. Hit "Compare".
5. The formats are converted and compared and the output is displayed in the output window. A ratio is also computed and displayed.
6. If wished, copy the comparison output to the clipboard and/or save it to a file and/or save it as book with an suitable format (HTML or text).

If you want to compare other formats, repeat step 1 and hit the "Refresh formats" button. Then repeat steps 2 - 5.
The "Compare"-Dialog is modeless, what permits to move it around and touch the Calibre screen.

Planned Features:
-----------------
- Remove soft hyphens before conversion.

Limitations:
------------
- The converted formats are stored as strings in memory, so large formats may run out of memory.

Version History:
----------------

Spoiler:

Version 1.2.4 - 01-07-2024
- Fixing an typo in version 1.2.3 (causes an error, when selecting other output types than "HTML"; thanks to Zillion_).
Version 1.2.3 - 12-28-2023
- Check wether the pdf format is readable ((encrypted pdf's, pdf's with no text layer).
- Substitute different quotes and dashes characters with standard characters before diff (optional).
- Debug print optional.
Version 1.2.2 - 06-29-2023
- Disable buttons to save diff result until a result is generated (Thanks to Robert1a)
Version 1.2.1 - 03-23-2023
- Switch between context line processing by plugin or by Difflib
Version 1.2.0 - 03-22-2023
- Abort compare with message if convert has no result.
- Hide identical lines, but with the option to display a number of context lines. Closes enhancement request #1.)
Version 1.1.2 - 02-03-2023
- Adding double-quotes for the --sr1-search value: --sr1-search "(?m)^\s*$"
to avoid "syntax error near unexpected token \`('" on Mac. (Thanks to irinel-dan.)
Version 1.1.1 - 11-30-2022
- Handle save file dialog with no user path/file choice.
Version 1.1.0 - 11-26-2022
- Changed tool button behavior: show compare dialog when icon clicked, show menu when arrow clicked (thanks to Comfy.n)
- Inverting HTML/CSS back colors (highlighting diffs) in dark mode (thanks to Comfy.n and Kovidgoyal)
Version 1.0.0 11-20-2022
- Initial release.

Installation:
-------------
Download the attached zip file and install the plugin as described in the plugins thread on mobileread.
You need to add the calibre path to your $PATH variable.

To report Bugs and suggestions:
-------------------------------
If you find any issues or have suggestions, please report them on GitHub or in the MobileRead Forum.

---

[GUI-Plugin] TextDiff - Version 1.2.4 - 01-07-2024

Ein Calibre GUI-Plugin zum Finden von Textunterschieden in zwei Buchformaten.

Haupteigenschaften:
-------------------
Dieses Plugin zeigt die Unterschiede zwischen zwei ausgewählten Buchformaten.
Die Formate wurden zunächst in Textformat konvertiert (auch wenn das Ausgangsformat bereits Text ist).
Wenn die Konvertierung fehlschlägt, kann das daran liegen, dass das Format keinen text enthält (wie z. B. bei gescannten PDF-Dateien) oder Calibre ein Konvertierungstool nicht finden kann (wie z. B. Microsoft wordconv).
Dann werden die Textdateien in den Speicher eingelesen und eventuell manipuliert (Leerzeilen und Ähnliches entfernen, wie unter "Geplante Features" beschrieben).
mit dem Konvertierungsprogramm von Calibre (https://manual.calibre-ebook.com/gen...k-convert.html).
Dann wird der Vergleich mit Pythons DiffLib (https://docs.python.org/3/library/difflib.html) durchgeführt.
Das Verhältnis gibt ein Maß für die Ähnlichkeit der beiden Texte an. 1,0 bedeutet, dass die Texte identisch sind, ein Wert nahe 0,0 bedeutet, dass die Texte völlig unterschiedlich sind.
Letzteres kann auch passieren, wenn das Quellformat keinen Textinhalt hat (wie gescannte PDF-Dateien). Dann sollte man ein neues Buch-Format (Text) mit einem OCR-Prozess erzeugen.

Der detaillierte Arbeitsablauf ist wie folgt:
1. Wählen Sie ein Buch mit mindestens zwei Formaten zum Vergleichen oder zwei Bücher mit jeweils mindestens einem Format aus.
2. Wählen Sie zwei Formate aus.
3. Wählen Sie das Ausgabeformat und andere Vergleichsoptionen.
4. Klicken Sie auf "Vergleichen".
5. Die Formate werden konvertiert und verglichen und die Ausgabe wird im Ausgabefenster angezeigt. Ein Verhältnis wird ebenfalls berechnet und angezeigt.
6. Falls gewünscht, kopieren Sie die Vergleichsausgabe in die Zwischenablage und/oder speichern Sie sie in einer Datei und/oder speichern Sie sie als Buch in einem geeigneten Format (HTML oder Text).

Wenn Sie andere Formate vergleichen möchten, wiederholen Sie Schritt 1 und klicken Sie auf die Schaltfläche "Formate aktualisieren".
Der "Vergleichen"-Dialog ist moduslos, was es erlaubt, ihn zu verschieben, und den darunterliegenden Calibre-Bildschirm zu steuern.

Geplante Funktionen:
--------------------
- Weiche Bindestriche vor der Konvertierung entfernen.

Einschränkungen:
----------------
- Die konvertierten Formate werden als Strings im Speicher gehalten, daher kann es bei großen Formaten zu Speichermangel kommen.

Versionsgeschichte:
-------------------

Spoiler:

Version 1.2.4 - 01-07-2024
- Behebung eines Tippfehlers in Version 1.2.3 (führte zu einem Fehleror, wenn andere Ausgabeformen als "HTML" gewählt wurden; Danke an Zillion_).
Version 1.2.3 - 12-28-2023
- Prüfung, ob das PDF-Format lesbar ist (kennwortgeschützt, gescannt).
- Optional: Ersatz von Zeichen vor dem Vergleich (Anführungszeichen, Striche, ...) durch Standardwerte.
- Optional: Testausgabe.
Version 1.2.2 - 06-29-2023
- Schaltflächen zum Speichern des Vergleichsergebnisses deaktiviert, bis Vergleichsergebnis erstellt wurde (Dank an Robert1a).
Version 1.2.1 - 03-23-2023
- Wahl zwischen Kontext-Verarbeitung durch Plugin oder Difflib.
Version 1.2.0 - 03-22-2023
- Abbruch mit Fehlermeldung, wenn mindestens eine Konvertierung in das Textformat fehlschlägt.
- Verbergen identischer Zeilen mit der Option, n Kontext-Zeilen darzustellen.
Version 1.1.2 - 02-03-2023
- Doppelte Anführungszeichen hinzugefügt für den Wert des Parameter --sr1-search value: --sr1-search "(?m)^\s*$"
zur Vermeidung der Meldung "syntax error near unexpected token \`('" auf dem Mac. (Danke an irinel-dan.)
Version 1.1.1 - 11-30-2022
- Abfangen im "Datei speichern"-Dialog, dass kein Format ausgewählt.
Version 1.1.0 - 26.11.2022
- Geändertes Verhalten der Werkzeugschaltfläche: Vergleichsdialog anzeigen, wenn auf das Symbol geklickt wird, Menü anzeigen, wenn auf den Pfeil geklickt wird (Dank an Comfy.n)
- Invertieren von HTML/CSS-Hintergrundfarben (Hervorheben von Unterschieden) im Dunkelmodus (Dank an Comfy.n und Kovidgoyal)
Version 1.0.0 20.11.2022
Erstveröffentlichung.

Installation:
-------------
Laden Sie die angehängte ZIP-Datei herunter und installieren Sie das Plugin wie im Thread "Einführung in Plugins" auf mobileread beschrieben.
Vergessen Sie nicht, Calibre in Ihre PATH-Variable aufzunehmen.

So melden Sie Fehler und Vorschläge:
------------------------------------
Wenn Sie Probleme finden oder Vorschläge haben, melden Sie diese bitte auf GitHub oder im MobileRead-Forum.

[JSWolf]

Originally Posted by feuille

Planned Features:
-----------------
- Custom characters to ignore ("char junk", e. g. "" vs. »«)

I hope when this is implemented that there will be an option to turn it off. I do want to know if the quotes are different.

Limitations:
------------
- The converted formats are stored as strings in memory, so large formats may run out of memory.

This does need to be addressed as some computers are only 32-bit or may not have enough memory. IMHO, store both text files on disk and compare that way instead of fully loading both text files into memory and then comparing.

Overall, I do like the idea of this plugin. Thanks.

[ownedbycats]

Originally Posted by JSWolf

I hope when this is implemented that there will be an option to turn it off. I do want to know if the quotes are different.

"Custom" seems to imply it'll be user-defined?

But yes, the memory limitation seems... not great. I'd use temp files.

This'll be a useful plugin though

[JSWolf]

Originally Posted by ownedbycats

"Custom" seems to imply it'll be user-defined?

But yes, the memory limitation seems... not great. I'd use temp files.

This'll be a useful plugin though

My laptop does have 16GB, but most don't. My Surface Pro 2 is only 8GB.

I do think the memory usage should be fixed right away.

[chaley]

Originally Posted by ownedbycats

But yes, the memory limitation seems... not great. I'd use temp files.

Originally Posted by JSWolf

My laptop does have 16GB, but most don't. My Surface Pro 2 is only 8GB.

I do think the memory usage should be fixed right away.

The plugin is keeping TXT files in memory. According to this thread on Quora, 1 GB is approximately 900,000 pages, or 179 million "standard" words (5 single-byte characters plus a space). If the text is 100% UTF-8 extended characters then using the same assumptions as in the thread there are 11 bytes per word. A GB would be 98 million words or 488,000 pages.

I don't know how many people will be comparing 900,000 page books, or even 488,000 page books.

[feuille]

Originally Posted by JSWolf

I hope when this is implemented that there will be an option to turn it off. I do want to know if the quotes are different.

There will be a option to turn it on

[JSWolf]

Originally Posted by chaley

The plugin is keeping TXT files in memory. According to this thread on Quora, 1 GB is approximately 900,000 pages, or 179 million "standard" words (5 single-byte characters plus a space). If the text is 100% UTF-8 extended characters then using the same assumptions as in the thread there are 11 bytes per word. A GB would be 98 million words or 488,000 pages.

I don't know how many people will be comparing 900,000 page books, or even 488,000 page books.

Do you think this could be a problem for a computer running a 32-bit version of Calibre where you only get A 3GB chunk to work with?

[ownedbycats]

Originally Posted by chaley

The plugin is keeping TXT files in memory. According to this thread on Quora, 1 GB is approximately 900,000 pages, or 179 million "standard" words (5 single-byte characters plus a space). If the text is 100% UTF-8 extended characters then using the same assumptions as in the thread there are 11 bytes per word. A GB would be 98 million words or 488,000 pages.

I don't know how many people will be comparing 900,000 page books, or even 488,000 page books.

Just curious: how would this work for PDFs?

While I probably wouldn't be running it specifically on those books, PDFs downloaded from the Internet Archive use some sort of layering compression that means that pdftotext can extract gigabytes of image layers into the temp folder alongside the text layer. (This happens when indexing for FTS or running the word count plugin, which should only require the text layer.) Would it try to keep all that in memory?

[chaley]

Originally Posted by JSWolf

Do you think this could be a problem for a computer running a 32-bit version of Calibre where you only get A 3GB chunk to work with?

Possibly, but a vanishingly small one. A user would need to be running a 32 bit version (not possible in calibre 6) and be comparing 2 books in excess of a few hundred thousand pages.

If I were the developer then doing the diffs using files would be low priority, especially if the underlying tool didn't directly support it.

FWIW: calibre keeps the entire db in memory.

@owndbycats: it converts to txt, which by definition contains only text. Where it gets the text from a layered pdf is another issue.

[ownedbycats]

Yeah, just mentioned it because other things that I thought should be text-only (full-text indexing and the Count Pages plugin) I've seen it extracting the images into temp.

[JSWolf]

I've given TextDiff a try and it's pretty good. Thanks for creating this plugin.

[feuille]

Originally Posted by JSWolf

Thanks for creating this plugin.

You're welcome

[Comfy.n]

Thank you for this plugin! I plan to use it more, in the future, to compare Orwell's 1984 translations, but I just gave it a try for testing purposes and noticed the HTML output to file will be great for this use case, as it displays the text differences side-by-side in a very synchronized fashion.

some quick notes:

• The toolbar button click could default directly to the "Compare" action, instead of triggering the menu opening, IMO...
  
• Calibre UI will hang for a while during TextDiff, so at first I wasn't sure that this was just a temporary halt.
  
• The "Save output as ebook" action sometimes returns an error, I forgot to take note of what it was about
  
• Under Dark theme, it's best to use the "Save output to file", as the on-screen output isn't tuned to that color palette

[feuille]

Thank you Comfy.n for the hints! I'll try to fix that. For a better feedback of the program status I had already considered a progress bar.

[feuille]

Hello Comfy.n,

I implemented two of your hints (see history).

In order to calculate the display time of a progress bar, I stopped intermediate times in the program flow. To my surprise, the HTML rendering of the text browser widget consumes 2/3 of the total runtime! I may implement in a future release a workaround like stepwise asynchronous loading with a timer or so, but I need to do some research on that.