[GUI Plugin] WordDumb

[GentleBook]

Hello everybody! I apologize if this question has been already asked, I searched inside the thread and did not find any correspondence.

I have portable caliber updated to last version, python 3.10.9 and obviously worddumb plugin! Recently often I got errors during x-raying ebooks (azw3 file type)

This is all the error message:

"Starting job: Generating X-Ray for *************
Job: "Generating X-Ray for ************" failed with error:
Traceback (most recent call last):
File "calibre\gui2\threaded_jobs.py", line 82, in start_work
File "calibre_plugins.worddumb.parse_job", line 119, in do_job
File "calibre_plugins.worddumb.deps", line 58, in install_deps
File "calibre_plugins.worddumb.deps", line 167, in pip_install
File "calibre_plugins.worddumb.utils", line 50, in run_subprocess
File "subprocess.py", line 524, in run
subprocess.CalledProcessError: Command '['py', '-m', 'pip', '--disable-pip-version-check', 'install', '-U', '-t', 'D:\\PORTATILE\\Calibre Portable\\Calibre Settings\\plugins\\worddumb-libs-py3.10', '--no-user', 'cupy-wheel==11.4.0']' returned non-zero exit status 1.

Called with args: ((3121, 'AZW3', 'D:\\PORTATILE\\Calibre Portable\\Calibre Library\\*********\\********* (3121)\\************.azw3', <calibre.ebooks.metadata.book.base.Metadata object at 0x000001BFDA70E770>, {'spacy': 'en_core_web_', 'wiki': 'en', 'kaikki': 'English', 'gloss': True, 'has_trf': True}), False, True) {'notifications': <queue.Queue object at 0x000001BFDA70E680>, 'abort': <threading.Event object at 0x000001BFCD75D510>, 'log': <calibre.utils.logging.GUILog object at 0x000001BFCD75D1E0>}
error: subprocess-exited-with-error

python setup.py egg_info did not run successfully.
exit code: 1

[26 lines of output]
[cupy-wheel] CUDA_PATH is not set.cupy-wheel may not be able to discover NVRTC to probe version
[cupy-wheel] Trying to detect CUDA version from libraries: ['nvrtc64_112_0.dll', 'nvrtc64_111_0.dll', 'nvrtc64_110_0.dll', 'nvrtc64_102_0.dll']
[cupy-wheel] Looking for library: nvrtc64_112_0.dll
[cupy-wheel] Failed to open nvrtc64_112_0.dll: Could not find module 'nvrtc64_112_0.dll' (or one of its dependencies). Try using the full path with constructor syntax.
[cupy-wheel] Looking for library: nvrtc64_111_0.dll
[cupy-wheel] Failed to open nvrtc64_111_0.dll: Could not find module 'nvrtc64_111_0.dll' (or one of its dependencies). Try using the full path with constructor syntax.
[cupy-wheel] Looking for library: nvrtc64_110_0.dll
[cupy-wheel] Failed to open nvrtc64_110_0.dll: Could not find module 'nvrtc64_110_0.dll' (or one of its dependencies). Try using the full path with constructor syntax.
[cupy-wheel] Looking for library: nvrtc64_102_0.dll
[cupy-wheel] Failed to open nvrtc64_102_0.dll: Could not find module 'nvrtc64_102_0.dll' (or one of its dependencies). Try using the full path with constructor syntax.
[cupy-wheel] No more candidate library to find
[cupy-wheel] ROCm detection unsupported on platform: win32
Traceback (most recent call last):
File "<string>", line 2, in <module>
File "<pip-setuptools-caller>", line 34, in <module>
File "C:\Users\lucam\AppData\Local\Temp\pip-install-puw4pqb6\cupy-wheel_8f72cdd3352d4fe080d332a6b280b3f5\setup.py", line 295, in <module>
main()
File "C:\Users\lucam\AppData\Local\Temp\pip-install-puw4pqb6\cupy-wheel_8f72cdd3352d4fe080d332a6b280b3f5\setup.py", line 276, in main
package = infer_best_package()
File "C:\Users\lucam\AppData\Local\Temp\pip-install-puw4pqb6\cupy-wheel_8f72cdd3352d4fe080d332a6b280b3f5\setup.py", line 252, in infer_best_package
raise AutoDetectionFailed(
__main__.AutoDetectionFailed:
================================================== ==========
Unable to detect NVIDIA CUDA or AMD ROCm installation.
================================================== ==========

[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

Encountered error while generating package metadata.

See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.


I have installed latest nvidia drivers.

Could some good soul help me ?

Thank you..

[xxyzz]

Originally Posted by GentleBook

================================================== ==========
Unable to detect NVIDIA CUDA or AMD ROCm installation.
================================================== ==========

Hi, you need to install CUDA(https://developer.nvidia.com/cuda-downloads) to use the "Run spaCy with GPU" feature.

[xxyzz]

Word Wise for non-English Kindle books feature is added to the master branch, please download from GitHub Actions Artifacts(https://github.com/xxyzz/WordDumb/ac...lows/tests.yml) to test this new feature.

The loading speed of customize Word Wise table is improved, and the table can be filtered by lemma, difficulty and enabled columns.

The translations of lemmas are not perfect but you can double click the text to edit the lemma column for non-English tables.

[domiten]

Thank you for that push! its exactly what I was looking for (well the next level would be having the gloss in another language too )

I did get this error when I tried to run it a second time

Starting job: Saving customized lemmas
Job: "Saving customized lemmas" failed with error:
Traceback (most recent call last):
File "calibre\gui2\threaded_jobs.py", line 82, in start_work
File "calibre_plugins.worddumb.config", line 332, in dump_lemmas_job
File "calibre_plugins.worddumb.utils", line 29, in run_subprocess
File "subprocess.py", line 501, in run
File "subprocess.py", line 966, in __init__
File "subprocess.py", line 1375, in _execute_child
File "subprocess.py", line 561, in list2cmdline
File "os.py", line 822, in fsdecode
TypeError: expected str, bytes or os.PathLike object, not tuple

Called with args: (True, WindowsPath('C:/Users/latob/AppData/Roaming/calibre/plugins/worddumb-lemmas/fr/kindle_fr_en_v0.db'), WindowsPath('C:/Users/latob/AppData/Roaming/calibre/plugins/worddumb-lemmas/fr/kindle_fr_en_dump_v0'), 'fr', 'en') {'notifications': <queue.Queue object at 0x000001B371BBE200>, 'abort': <threading.Event object at 0x000001B371BBC640>, 'log': <calibre.utils.logging.GUILog object at 0x000001B371BBEB30>}

Any time I changed the lemma language under "Customize Kindle Word Wise" or "Customize EPUB Wiktionary" after the first time I ran it; it keeps defaulting to catalan and not saving the state of the new language. The above error also appears

[xxyzz]

Originally Posted by domiten

(well the next level would be having the gloss in another language too )

Kindle supports Chinese definitions for Word Wise. For other languages that would require parsing non-English Wiktionary dump files and probably need to reverse-engineer some columns of the Word Wise klld database. Both are not trivial tasks.

Originally Posted by domiten

Any time I changed the lemma language under "Customize Kindle Word Wise" or "Customize EPUB Wiktionary" after the first time I ran it; it keeps defaulting to catalan and not saving the state of the new language. The above error also appears

This error is fixed on the master branch, last selected lemma language is also remembered. Please download the latest zip file from GitHub Actions.

[medinauta]

Is there a way to make X-Ray check the Fandom for tv pages instead of landing on the disambiguation page with the tv show?

The Wikipedia results are so random with most names not found and the few it does have no relation with the book or are mis recognized by spaCy (CUDA is not an option for me since my pc is Intel).

Any advice?

[medinauta]

Originally Posted by medinauta

Is there a way to make X-Ray check the Fandom for tv pages instead of landing on the disambiguation page with the tv show?

The Wikipedia results are so random with most names not found and the few it does have no relation with the book or are mis recognized by spaCy (CUDA is not an option for me since my pc is Intel).

Any advice?

https://imgur.com/NJO2w8W[/IMG]

https://imgur.com/V62N6qD

[xxyzz]

Originally Posted by medinauta

Any advice?

I pushed a commit https://github.com/xxyzz/WordDumb/co...11826f6ae82c6e to the master branch which will choose the most similar title in a Fandom disambiguation page. In this case , the "Amos Burton (Books)" page is used. And you also have to delete the old Fandom cache file in the "plugins/worddumb-fandom" folder(it's in the calibre configuration folder).

You can also use the "Customize X-Ray" feature to add character descriptions manually.

The test plugin zip file can be downloaded at GitHub: https://github.com/xxyzz/WordDumb/ac...uns/4084130619

The master branch also has a new feature: finding Word Wise definition using the POS type.

[PoP]

Originally Posted by xxyzz

I guess you mean enable Word Wise on non-English books, right?
It's possible.

Originally Posted by xxyzz

Word Wise for non-English Kindle books feature is added to the master branch

Here is my experiment of a book in german (which I can't read):

Originally Posted by domiten

Thank you for that push! its exactly what I was looking for (well the next level would be having the gloss in another language too )

As explained by @xxyzz on Github. Can be done! Limited to few languages, and only for some specific Kindle dictionaries. Here is the above experiment with the gloss in french:

Originally Posted by xxyzz

Kindle supports Chinese definitions for Word Wise. For other languages that would require parsing non-English Wiktionary dump files and probably need to reverse-engineer some columns of the Word Wise klld database. Both are not trivial tasks.

ACK. Sourcing from Kindle dictionary is far from usable. My experiment won't lead me to read in german anytime soon

Wikitionary is so much richer ...and unfortunately complex.

[medinauta]

Originally Posted by xxyzz

The test plugin zip file can be downloaded at GitHub: https://github.com/xxyzz/WordDumb/ac...uns/4084130619

Sorry but how do I download the plug-in zip file from there?

[domiten]

Originally Posted by PoP

Here is my experiment of a book in german (which I can't read):
Attachment 199456



As explained by @xxyzz on Github. Can be done! Limited to few languages, and only for some specific Kindle dictionaries. Here is the above experiment with the gloss in french:
Attachment 199457




ACK. Sourcing from Kindle dictionary is far from usable. My experiment won't lead me to read in german anytime soon

Wikitionary is so much richer ...and unfortunately complex.

I know I'm asking for a lot, but is there any way to walk me/us through creating the gloss for other languages? I wasn't able to figure out how to do it by looking through the READMEs in github. My target is gloss in French for what its worth. I appreciate any help!

[PoP]

Originally Posted by domiten

I know I'm asking for a lot, but is there any way to walk me/us through creating the gloss for other languages? I wasn't able to figure out how to do it by looking through the READMEs in github. My target is gloss in French for what its worth. I appreciate any help!

The instructions which I referenced from post #492 have been removed from the master branch by xxyzz.

I didn't ask xxyzz for permission but attached is a copy of his (now defunct) directory with instructions and code.

I presume 1) the results were not to his satisfaction 2) this method was too limited 3) he didn't want to maintain that code 4) he is contemplating the Wiktionary non-trivial approach in the future.

Asking for forgiveness, @xxyzz please advise if you want me to remove the attachment.

[xxyzz]

Originally Posted by medinauta

Sorry but how do I download the plug-in zip file from there?

You can download the test zip file at the "Artifacts" card, a GitHub account is required though. If you don't have one, here is the latest build.

[xxyzz]

Originally Posted by PoP

please advise if you want me to remove the attachment.

It's fine. In fact, the code still can be found in the git log. As you mentioned, the quality of the created dictionary file is not very good. And the code in that directory hasn't changed in a long time, it's already unmaintained.

[xxyzz]

I finally figure out how to create Word Wise klld database, turns out those unknown columns doesn't matter. I have updated the Proficiency and WordDumb code to use newly created klld files from Wiktionary data for Kindle books.

Add support of other gloss languages now seems more feasible. I found this cool project: dbnary, they parses Wiktionary dump files in several languages and provides Turtle files which can be queried in RDF.