[GUI Plugin] WordDumb

[xxyzz]

New test version

• Send files to Android via adb, please read the document at GitHub
• Improve X-Ray and Word Wise quality:
  • Merge person partial name X-Ray entities
  • Unescape HTML text
  • Change fuzz algorithm
  • Add 149 and update 53 lemmas
  • Remove WORK_OF_ART NER label
  • Remove cardinal directions and ordinal directions X-Ray entities
  • Remove whitespaces from book quote sentence
  • Add option to enable or disable EPUB locator map
  • Insert X-Ray image caption data

Commits: https://github.com/xxyzz/WordDumb/co....19.0...master

The latest plugin will be uploaded to GitHub Actions Artifacts at each git push automatically.

[xxyzz]

v3.20.0

New features:

• Send files to Android device via adb
• Customize Word Wise lemmas
• Insert X-Ray image caption data
• Add an option to enable or disable EPUB locator map

Bug fixes:

• Fix the code that should check ASIN of books on Kindle but check books in library
• Disable pip cache to fix memory error on some devices
• Fix a typo in dump_lemmas.py

Other changes:

• Merge person X-Ray entities when partial name appears first
• Merge X-Ray entities that get redirected or normalized by MedaiWiki API
• Unescape MOBI and EPUB HTML text
• Only use texts inside the MOBI and EPUB body tag
• Select orthographic locator map in SPARQL query
• Add lemmas that contain parentheses
• Change X-Ray fuzz algorithm to token_set_ratio()
• Remove WORK_OF_ART NER label
• Remove cardinal directions and ordinal directions X-Ray entities
• Add 639 and update 173 lemmas

Looks like spaCy will release v3.3.0 soon.

[xxyzz]

v3.20.1

• Add support of Finnish, Korean and Swedish language
• Update spaCy to 3.3.0
• Disable jobs_pointer

[Magic815]

Kind of a dumb question, but since I see that WordDumb handles the creation of the files and sends the ebook to my Kindle device, I wanted to confirm something.

My typical workflow is to ingest EPUB files in Calibre, and then I have them auto-convert to KFX when I use the 'Send to device' button. So does that mean that I need to previously have a KFX file already made, when I click the WordDumb button? I wasn't sure if sending via WordDumb would also handle the same auto-conversion that occurs with the Send to device method in Calibre.

[xxyzz]

Originally Posted by Magic815

So does that mean that I need to previously have a KFX file already made, when I click the WordDumb button?

That's right. WordDumb doesn't convert book formats. It'll create a new EPUB file with X-Ray footnotes if a book only has EPUB format.

[muggleMode]

hi, thanks for this amazing plugin.

just a couple newbie questions
Does "Wordwise" only work in English?
Do you have a book where I can test all the functions working fine on a kindle pw5?

[xxyzz]

Originally Posted by muggleMode

Does "Wordwise" only work in English?

Yes. It might be possible to create a new dictionary file and set the book language to English to enable Word Wise for other languages.

Originally Posted by muggleMode

Do you have a book where I can test all the functions working fine on a kindle pw5?

https://standardebooks.org/ebooks

[Magic815]

Hi there - I had a follow-up question about this plugin. Let's say I have already converted an EPUB file to a KFX file - and it's the KFX file that I want to send to my eReader device.

When I hit the WordDumb plugin on that eBook, is WordDumb actually modifying the KFX file that is stored in my Calibre library permanently - and then it sends it over to my eReader? Or is WordDumb modifying a copy of the KFX file and sending that copy to the eReader device, and the KFX file in my library stays untouched? Or is WordDumb sending over extra files that sit alongside the KFX file on my eReader as it's way of embedding that information?

[xxyzz]

Both Word Wise and X-Ray data are stored in a separate sqlite file. WordDumb only adds a random ASIN metedata to the book if the book doesn't have this data, otherwise it won't alter the book file.

[Magic815]

Originally Posted by xxyzz

Both Word Wise and X-Ray data are stored in a separate sqlite file. WordDumb only adds a random ASIN metedata to the book if the book doesn't have this data, otherwise it won't alter the book file.

Interesting. Where does that separate sqlite file get stored/reside? Does it move to the eReader and sit next to the KFX file? As for the IDs, so you're saying if my workflow always ensures that I have ISBN, ASIN, and GoodReads ID information in my IDs metadata field, WordDumb will not touch the KFX file itself in any way?

[xxyzz]

Originally Posted by Magic815

Where does that separate sqlite file get stored/reside?

It's located at the same folder of the book file before being moved to your device.

Originally Posted by Magic815

Does it move to the eReader and sit next to the KFX file?

It's sent to a "book_name.sdr" folder next to the book file.

Originally Posted by Magic815

As for the IDs, so you're saying if my workflow always ensures that I have ISBN, ASIN, and GoodReads ID information in my IDs metadata field, WordDumb will not touch the KFX file itself in any way?

As long as the book file has a valid ASIN metadata, WordDumb won't alter the file.

[xxyzz]

v3.21.0

New features

• Add option to adjust preferred book format order
• Add option to create files for all available book format

Improve X-Ray quality

• Thanks to jhowell, KFX footnote reference numbers are ignored(requires KFX input 1.49.0)
• Don't merge different person with the same given name or surname
• Increase KFX image caption length limit to 450
• Ignore URL links and page number references
• Delete " of" from the end of x-ray label

Other changes

• Convert NER label constant variables to frozenset
• Only save MediaWiki cache if new data is added
• Update dependencies

[xxyzz]

v3.21.1

• Disable pip version check
• Fix PY_PATH undefined error on macOS

[xxyzz]

v3.21.2

Bug fixes

• Fix media_type undefined error when adding locator map image to EPUB footnote
• Fix Fandom cache folder not exists error
• Fix substring not found error for some MOBI , AZW3 and EPUB books

Other changes

• Replace full name in Chinese and Japanese books that have interpunct
• Only extent X-Ray length when the next character is punctuation
• Update dependencies

[j.p.s]

Thank you for making this plugin.

Does WordDumb have an option for the user to supply descriptions for x-ray entities?