[GUI Plugin] WordDumb

[samuelj]

Hi,

Just a quick question, how do I use x-ray? Do I have to do some configuration for each book, because I checked it with "A game of thrones" book and I have found that some of the names didn't have any x-ray information.

Really am thankful for your efforts.

[xxyzz]

Originally Posted by samuelj

Do I have to do some configuration for each book, because I checked it with "A game of thrones" book and I have found that some of the names didn't have any x-ray information.

There are usually two cases why some names are not included:
1: Wikipedia or Fandom doesn't have a page for that character
2: spaCy couldn't recognize the name as a name

You could use the "Customize X-Ray" feature to tell spaCy to find the name and add a description if Wikipedia or Fandom doesn't have data for that character.

And this configuration file will be saved at the same folder of the book file. You can copy this file to other book folders.

[xxyzz]

v3.28.0

New features:

• Use MediaWiki's parse API to get text for some Wikipedia titles redirected to sections
• Remove references and error messages from parsed Fandom text
• Separate X-Ray paragraphs in <p> tags for EPUB footnotes
• Allow "-" in Fandom URL's language code part
• Install cupy-wheel package instead of setting cupy version manually

Bug fixes:

• Replace %xx escapes in EPUB opf href strings
• Fix error when creating X-Ray file and customize X-Ray data is provided
• Fix pip install error caused by space in user name
• Install compiled with CUDA enbabled PyTorch package for Windows and Linux
• Fix None reference error in transformers library
• Fix lxml not found error
• Fix UnicodeDecodeError when open JSON files on Windows
• Check notifications variable is not None before using it

Other changes:

• Add back downloading Wiktionary file notification
• Update Proficiency v0.4.0
• Update rapidfuzz 2.13.3

[PoP]

@xxyzz Thanks for all the work.

Just for fun, when X-RAYing a typical 350 pages book with WordDumb
On my system: Windows 10 64bits & AMD Ryzen 7 3800X & NVIDIA GeForce GTX 1660
CPU model: elapsed time 12sec GPU@2% CPU@12%
GPU model: elapsed time 77sec GPU@90% CPU@3%

HaHaHa, I would have thought otherwise. Shows how poor my rig is at gaming!

[xxyzz]

Maybe spaCy finds some new X-Ray entities with GPU model and queries them on Wikipedia API takes some time at the first time, create X-Ray for the same book later would use a Wikipedia cache file.

Memory usage could also be a factor. I don't know how much RAM(system and GPU RAM) spaCy GPU model would use, but I guess 6G should be enough.

And spaCy doc says the GPU model has higher NER accuracy, do you think the X-Ray file get better?

[PoP]

^ Good points, but I took the measurements on a second round, after any data had been cached (I presume).

> Do you think the X-Ray file get better

On a cursory look, it is not much different. For my sample book,

The XRAY.entities.BB3J0DJUK4.asc CPU version is 458,752 bytes

The XRAY.entities.BB3J0DJUK4.asc GPU version is 430,080 bytes

Go figure :-)

[xxyzz]

For "better" I mean is the GPU model "smarter" than the CPU model. For instance, does the GPU model produces less nonsense X-Ray names or can it detect names that CPU model can't find?

I remember running the transformers model on CPU once before(slow as hell) and both X-Ray files don't have much difference.

[PoP]

Originally Posted by xxyzz

For "better" I mean is the GPU model "smarter" than the CPU model. For instance, does the GPU model produces less nonsense X-Ray names or can it detect names that CPU model can't find?

Very subjectively, It doesn't appear smarter to me, it is marginally different and remains as useful. No blatant nonsense in either version.

Judge yourself by comparing these CPU/GPU screenshots:

Peoples:




Places:


Images:


[EDIT 18:42] the The chatGPT Bot seems to think so:

Spoiler:

Is spacy cuda model smarter and more accurate than the cpu model?

Yes, the CUDA version of spaCy is generally faster and more accurate than the CPU version because it can take advantage of the parallel processing capabilities of a CUDA-enabled GPU. This can lead to significant performance improvements for certain types of operations, especially when working with large amounts of text data. However, the CPU version of spaCy can still be a good choice for certain scenarios, such as when using a machine that does not have a CUDA-enabled GPU or when the amount of data being processed is not large enough to benefit from the additional performance of the CUDA version. It ultimately depends on your specific use case and requirements.

[Taubin]

Edit: Nevermind, I forgot to hit save, sorry.

[xxyzz]

v3.28.1

• Add "requires CUDA" text to "run spaCy with GPU" checkbox
• Update French translation texts by @miiPoP in #94
• Update dependencies, fix pip install error on macOS
• Add PyTorch version

[xxyzz]

v3.28.2

• Fix install thinc-apple-ops package breaks dependencies bug on ARM macOS
• Update rapidfuzz 2.13.7

[sonycrank]

Hey, do i need to install/download the language packs from spacy manually?

I don't know why but my X-RAY doesn't work right. If i click on a name of a character my kindle shows only some quotes and mentions of this character but no wiki or fandom.

I tried this with https://lotr.fandom.com/de and https://gameofthrones.fandom.com/ru

[xxyzz]

Originally Posted by sonycrank

Hey, do i need to install/download the language packs from spacy manually?

No, you don't. spaCy model is downloaded automatically for the book language.

Originally Posted by sonycrank

I don't know why but my X-RAY doesn't work right. If i click on a name of a character my kindle shows only some quotes and mentions of this character but no wiki or fandom.

I tried this with https://lotr.fandom.com/de and https://gameofthrones.fandom.com/ru

Probably the Fandom site doesn't have this character page, maybe the book and the Fandom site use different languages. Could you find the character in these Fandom sites or Wikipedia?

And Wikipedia won't be used if a Fandom link is provided.

[domiten]

Hi @XXYYZZ, first of all thank you for all the work on WordDumb, Its been great to enhance some of my books.

I wanted to ask you if you think there will ever be a way to enable WordWise on English books (or trick kindle to thinking a book in another language is in English) and display wordwise on it?

I'm working on another language, and it would be a huge boost to get it enabled.

Thanks!

[xxyzz]

Originally Posted by domiten

I wanted to ask you if you think there will ever be a way to enable WordWise on English books (or trick kindle to thinking a book in another language is in English) and display wordwise on it?

I guess you mean enable Word Wise on non-English books, right?

It's possible.

First change the book medate: For MOBI and AZW3 books, the locale MOBI header need to be "9" and the language EXTH header set to "en".

Then create the word wise db file, this requires mapping the new language words to English words. I could get the translation data from Wiktionary.

There is one minor drawback, the default fonts would be English fonts. I guess you can get around this with custom font.

Here is a demo screenshot of a Chinese book: