[GUI Plugin] WordDumb

[j.p.s]

I have not tried the latest plugin yet, but it sounds like it is exactly what I need. Thank you!

Can you post an example of running the plugin from a terminal?

[xxyzz]

I just pushed the cli support commit to GitHub. You can run this command to create files.

calibre-debug -r WordDumb -- book_path

Use `-h` to check other options:

calibre-debug -r WordDumb -- -h

[j.p.s]

Originally Posted by xxyzz

Have you tried the latest plugin from GitHub Action?

I tried it and it worked.

Thanks!

[j.p.s]

Originally Posted by xxyzz

I just pushed the cli support commit to GitHub. You can run this command to create files.

calibre-debug -r WordDumb -- book_path

Use `-h` to check other options:

calibre-debug -r WordDumb -- -h

Thanks! I will give it a try.

Can the book path be someplace not in calibre's library? (Not added to calibre)

[xxyzz]

Originally Posted by j.p.s

Thanks! I will give it a try.

Can the book path be someplace not in calibre's library? (Not added to calibre)

Yes, the book file can be anywhere. But the language and format of the book must be supported.

[j.p.s]

Originally Posted by xxyzz

I just pushed the cli support commit to GitHub.

Originally Posted by xxyzz

Yes, the book file can be anywhere. But the language and format of the book must be supported.

It worked great. Using the CLI was much easier for me.

Unrelated to CLI, but I can also report that for a couple of books about Roman history, it did a better job for several of the most important (major historical) characters than the X-ray files supplied by amazon for the books.

Some people are designated terms and some terms are designated as people.

Also, one of the changes you made in past week or so also fixed a crash that I was going to report after the new functionality was in place.

[xxyzz]

Originally Posted by j.p.s

it did a better job for several of the most important (major historical) characters than the X-ray files supplied by amazon for the books.

I though Amazon's files are created manually...

Originally Posted by j.p.s

Some people are designated terms and some terms are designated as people.

If you have a CUDA-compatible GPU, you could try spaCy's transformer model(https://spacy.io/usage#gpu), it has higher NER accuracy then the CPU model. You also have to install the dependencies manually and change the code a bit(a few lines, maybe).

Ultimately, you could train your own model for specificity kinds of books.

[j.p.s]

Originally Posted by xxyzz

I though Amazon's files are created manually...

They supposedly are made at least in part manually, but see my thread "How many X-ray mistaken identities can you find in your books?"
https://www.mobileread.com/forums/sh...d.php?t=309190
and "Easily fix egregious X-ray errors"
https://www.mobileread.com/forums/sh...d.php?t=303347
especially post #6, which is one of the books I tested WordDumb on.

If you have a CUDA-compatible GPU, you could try spaCy's transformer model(https://spacy.io/usage#gpu), it has higher NER accuracy then the CPU model. You also have to install the dependencies manually and change the code a bit(a few lines, maybe).

Ultimately, you could train your own model for specificity kinds of books.

I use the open source nvidia driver, so I can't use CUDA.

I'm almost out of time for this for a while, but I think I am almost ready to use it for a few books that don't have X-ray instead of testing it against books that do have X-ray.

[xxyzz]

Originally Posted by j.p.s

They supposedly are made at least in part manually, but see my thread "How many X-ray mistaken identities can you find in your books?"
https://www.mobileread.com/forums/sh...d.php?t=309190
and "Easily fix egregious X-ray errors"
https://www.mobileread.com/forums/sh...d.php?t=303347
especially post #6, which is one of the books I tested WordDumb on.

But the SQL files you shared only work for certain book files, if the publisher changes just one character the following X-Ray offsets are all changed. Maybe you could shared the corrections as customized X-Ray JSON file, so all formats of the same book can use this file and others can further improve it.

[j.p.s]

Originally Posted by xxyzz

But the SQL files you shared only work for certain book files, if the publisher changes just one character the following X-Ray offsets are all changed. Maybe you could shared the corrections as customized X-Ray JSON file, so all formats of the same book can use this file and others can further improve it.

I stopped working on that because it was too much work and no one showed any interest.

I am only referring to those threads as an example of how many bad errors are in almost every X-ray file supplied by amazon. I think the purpose of most X-ray generating utilities is to let people make their own X-ray files for books that do not have them, but I think WordDumb has the potential to make better X-ray files than amazon.

[xxyzz]

I added the source column to the customize X-Ray table. `None` means the description is a quote from the book, `1` for Wikipedia and `2` for Fandom.

[dizaam]

hello im stuck when running worddump on installing en_core_web_lg

[xxyzz]

Originally Posted by dizaam

hello im stuck when running worddump on installing en_core_web_lg

That size of that model is 382 MB(https://spacy.io/models/en#en_core_web_lg). You could select the medium model(31 MB) if you don't want to wait.

[xxyzz]

v3.22.0

New features

• Use customize X-Ray entities to fix NER errors
• Add command line interface
• Include POS type to customize Word Wise table
• Add GPE entity founding date to X-Ray description
• Add X-Ray entity minimal occurrences option

Bug fixes

• Fix upper case image file not found error
• Only copy Word Wise dictionary file from device
• Fix PyQt6 Enum errors

[j.p.s]

When a (first?) occurrance of an X-ray entity with 2 or more words has a newline between 2 of the words in the text of the book the entity label has a newline between those words in the generated X-ray file. This can be seen in the attached example.png

WordDumb might also make two entiies, one with a newline embedded in the label and one without. It is also possible to workaround this by making an entry in the worddumb-custom-x-ray.json file for the book with an alias that has "\n" in place of the space as can be seen in workaround.txt

Also attached is NewlineBug.tar.gz which has an azw3 of the public domain book, a worddumb-custom-x-ray.json for both cases, a temporary worddumb.json plugin configuration file, the 2 generated XRAY.*.asc files, and a script to generate the XRAY files and tsv dumps of the entity and entity_description tables for both cases.