[GUI Plugin] WordDumb

[xxyzz]

I finally figure out how to create Word Wise klld database, turns out those unknown columns doesn't matter. I have updated the Proficiency and WordDumb code to use newly created klld files from Wiktionary data for Kindle books.

Add support of other gloss languages now seems more feasible. I found this cool project: dbnary, they parses Wiktionary dump files in several languages and provides Turtle files which can be queried in RDF.

[PoP]

Originally Posted by PoP

... 4) he is contemplating the Wiktionary non-trivial approach in the future...

Originally Posted by xxyzz

I finally figure out how to create Word Wise klld database ... I have updated the Proficiency and WordDumb code to use newly created klld files from Wiktionary data for Kindle books.

Splendid!

[xxyzz]

New Word Wise definition languages are added, please download the latest test plugin from GitHub Actions.

WordDumb now supoorts 17 Word Wise gloss languages: German, Greek, English, Spanish, Finnish, French, Croatian, Italian, Japanese, Lithuanian, Dutch, Norwegian, Polish, Portuguese, Russian, Swedish, Chinese.

Some languages don't have infection data, "use POS type to find Word Wise definition" should be enabled for those languages.

[domiten]

Originally Posted by xxyzz

New Word Wise definition languages are added, please download the latest test plugin from GitHub Actions.

Amazing!

[xxyzz]

Hi folks, check out the new document website: https://xxyzz.github.io/WordDumb/.

[mrsidnaik]

Starting job: Generating Word Wise and X-Ray for Gardens of the Moon
Job: "Generating Word Wise and X-Ray for Gardens of the Moon" failed with error:
Traceback (most recent call last):
File "calibre\gui2\threaded_jobs.py", line 82, in start_work
File "calibre_plugins.worddumb.parse_job", line 167, in do_job
File "calibre_plugins.worddumb.utils", line 50, in run_subprocess
File "subprocess.py", line 524, in run
subprocess.CalledProcessError: Command '['python', 'C:\\Users\\ASUS\\AppData\\Roaming\\calibre\\plugi ns\\WordDumb.zip', '', 'C:\\Users\\ASUS\\Google Drive\\Calibre Library\\Steven Erikson\\Gardens of the Moon (9092)\\Gardens of the Moon - Steven Erikson.epub', '', '', 'en_core_web_trf', 'en', 'en', '', 'WordDumb/3.28.2 (https://github.com/xxyzz/WordDumb)', 'cn', '', 'EPUB', '1', 'C:\\Users\\ASUS\\AppData\\Roaming\\calibre\\plugi ns\\WordDumb.zip', '', '-l', '-x', '-m']' returned non-zero exit status 1.

Called with args: ((9092, 'EPUB', 'C:\\Users\\ASUS\\Google Drive\\Calibre Library\\Steven Erikson\\Gardens of the Moon (9092)\\Gardens of the Moon - Steven Erikson.epub', <calibre.ebooks.metadata.book.base.Metadata object at 0x00000136EAC49CF0>, {'spacy': 'en_core_web_', 'wiki': 'en', 'kaikki': 'English', 'gloss': True, 'has_trf': True}), True, True) {'notifications': <queue.Queue object at 0x00000136EAC49ED0>, 'abort': <threading.Event object at 0x00000136EAC48AC0>, 'log': <calibre.utils.logging.GUILog object at 0x00000136EAC498D0>}
Traceback (most recent call last):
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python. 3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python. 3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "C:\Users\ASUS\AppData\Roaming\calibre\plugins\Wor dDumb.zip\__main__.py", line 13, in <module>
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 664, in _load_unlocked
File "<frozen importlib._bootstrap>", line 627, in _load_backward_compatible
File "<frozen zipimport>", line 259, in load_module
File "C:\Users\ASUS\AppData\Roaming\calibre\plugins\Wor dDumb.zip\dump_kindle_lemmas.py", line 13, in <module>
TypeError: unsupported operand type(s) for |: 'type' and 'NoneType'

[xxyzz]

Originally Posted by mrsidnaik

PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64 __qbz5n2kfra8p0\lib\runpy.py", line 87, in _run_code
TypeError: unsupported operand type(s) for |: 'type' and 'NoneType'

Update Python version to 3.10 or above.

[xxyzz]

v3.29.0

New features:

• Support Word Wise for non-English Kindle books
• Support more Word Wise gloss languages
• Use POS type to find more accurate Word Wise defintion
• Significantly improve customized Word Wise table load speed
• Remember last opened customize lemmas language
• Improve Word Wise quality for non-English language books
• Use intro from the most similar title in Fandom disambiguation page
• Save Wikipedia and Fandom cache data in SQLite file
• Remove the word joiner characters from book text

Bug fixes:

• Compare the original word's length with the length of gloss
• Fix EPUB Word Wise footnotes tag mismatch error for Apple Books
• Raise exception if Python version is too old

Other changes:

• New document website: https://xxyzz.github.io/WordDumb
• Update fr.po by @miiPoP in #102 and #108

[domiten]

Updated to the latest release via Calibre. It looks like I'm getting errors on the X-Ray portion specifically for certain books - Running the Word Wise portion by itself avoids the issue (this was based on the last lines of the error being for the new SQLite3 tables, and spacy docs, making me think its X-Ray related). It doesn't happen for all books, I'm unable to consistently reproduce the error.

Starting job: Generating Word Wise and X-Ray for Harry Potter à l'école des sorciers
Job: "Generating Word Wise and X-Ray for Harry Potter à l'école des sorciers" failed with error:
Traceback (most recent call last):
File "calibre\gui2\threaded_jobs.py", line 82, in start_work
File "calibre_plugins.worddumb.parse_job", line 186, in do_job
File "calibre_plugins.worddumb.utils", line 51, in run_subprocess
File "subprocess.py", line 524, in run
subprocess.CalledProcessError: Command '['py', 'C:\\Users\\latob\\AppData\\Roaming\\calibre\\plug ins\\WordDumb.zip', '{"create_ww": true, "create_x": true, "asin": "BBJLF4M77W", "book_path": "C:\\\\Users\\\\latob\\\\Calibre Library\\\\J. K. Rowling\\\\Harry Potter a l\'ecole des sorciers (20)\\\\Harry Potter a l\'ecole des sorc - J. K. Rowling.mobi", "acr": "Harry_Potter_a_l\'ecole_des_sorc", "revision": "cf653006", "model": "en_core_web_md", "lemma_lang": "en", "mobi_codec": "utf-8", "useragent": "WordDumb/3.29.0 (https://github.com/xxyzz/WordDumb)", "book_fmt": "MOBI", "plugin_path": "C:\\\\Users\\\\latob\\\\AppData\\\\Roaming\\\\cal ibre\\\\plugins\\\\WordDumb.zip"}', '{"use_pos": true, "search_people": false, "model_size": "md", "zh_wiki_variant": "cn", "fandom": "", "add_locator_map": false, "preferred_formats": ["MOBI", "KFX", "EPUB", "AZW3", "AZW"], "use_all_formats": false, "minimal_x_ray_count": 1, "en_ipa": "ga_ipa", "zh_ipa": "pinyin", "choose_format_manually": true, "wiktionary_gloss_lang": "fr", "kindle_gloss_lang": "fr", "use_gpu": false, "cuda": "cu117", "last_opened_kindle_lemmas_language": "fr", "last_opened_wiktionary_lemmas_language": "fr", "use_wiktionary_for_kindle": false, "ca_wiktionary_difficulty_limit": 5, "da_wiktionary_difficulty_limit": 5, "de_wiktionary_difficulty_limit": 5, "el_wiktionary_difficulty_limit": 5, "en_wiktionary_difficulty_limit": 5, "es_wiktionary_difficulty_limit": 5, "fi_wiktionary_difficulty_limit": 5, "fr_wiktionary_difficulty_limit": 5, "hr_wiktionary_difficulty_limit": 5, "it_wiktionary_difficulty_limit": 5, "ja_wiktionary_difficulty_limit": 5, "ko_wiktionary_difficulty_limit": 5, "lt_wiktionary_difficulty_limit": 5, "mk_wiktionary_difficulty_limit": 5, "nl_wiktionary_difficulty_limit": 5, "no_wiktionary_difficulty_limit": 5, "pl_wiktionary_difficulty_limit": 5, "pt_wiktionary_difficulty_limit": 5, "ro_wiktionary_difficulty_limit": 5, "ru_wiktionary_difficulty_limit": 5, "sv_wiktionary_difficulty_limit": 5, "uk_wiktionary_difficulty_limit": 5, "zh_wiktionary_difficulty_limit": 5}']' returned non-zero exit status 1.

Called with args: ((20, 'MOBI', "C:\\Users\\latob\\Calibre Library\\J. K. Rowling\\Harry Potter a l'ecole des sorciers (20)\\Harry Potter a l'ecole des sorc - J. K. Rowling.mobi", <calibre.ebooks.metadata.book.base.Metadata object at 0x000001BFD920E020>, 'en'), True, True) {'notifications': <queue.Queue object at 0x000001BFD920DC90>, 'abort': <threading.Event object at 0x000001BFD920DF90>, 'log': <calibre.utils.logging.GUILog object at 0x000001BFD920DD50>}
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "C:\Users\latob\AppData\Roaming\calibre\plugins\Wo rdDumb.zip\__main__.py", line 40, in <module>
File "C:\Users\latob\AppData\Roaming\calibre\plugins\Wo rdDumb.zip\parse_job.py", line 245, in create_files
File "C:\Users\latob\AppData\Roaming\calibre\plugins\Wo rdDumb.zip\parse_job.py", line 782, in create_spacy_matcher
File "C:\Users\latob\AppData\Roaming\calibre\plugins\Wo rdDumb.zip\dump_lemmas.py", line 96, in save_spacy_docs
File "C:\Users\latob\AppData\Roaming\calibre\plugins\Wo rdDumb.zip\dump_lemmas.py", line 137, in create_lemma_patterns_with_pos
sqlite3.OperationalError: no such table: senses

[xxyzz]

Originally Posted by domiten

sqlite3.OperationalError: no such table: senses

The English Kindle Word Wise lemmas database file seems corrupted. Delete the "kindle_en_en_v0.db" in "/AppData/Roaming/calibre/plugins/worddumb-lemmas/en" folder then try again.

[MonkCanatella]

Hi! Love that you've made this program and would love for it to work, but I tried on the autobiography of Maya Angelou and it doesn't seem to work.

https://i.imgur.com/nKZMUGP.png

Should this be filled in with all the entries? I'm using iPad Pro with latest update and the latest update of the Kindle App for ios. I sideloaded the KFX using itunes for pc

[xxyzz]

Originally Posted by MonkCanatella

Should this be filled in with all the entries? I'm using iPad Pro with latest update and the latest update of the Kindle App for ios. I sideloaded the KFX using itunes for pc

That's for customizing X-Ray entries, you can overwrite names if you spot some incorrect entries.

You also have to sideload the generated X-Ray file, but I don't know how to do that on Apple devices.

[MonkCanatella]

Oh, so at the moment there's no known way to add x-ray to apple devices? How do I know if the wordwise/xray has populated correctly?

[j.p.s]

Originally Posted by MonkCanatella

Oh, so at the moment there's no known way to add x-ray to apple devices? How do I know if the wordwise/xray has populated correctly?

The xray file is an sqlite database. You could use something like sqlitebrowser to look at it.

Sorry, but I don't know anything about wordwise.

[emntsha]

hi, im trying for the first time to use wordwise. i have python and pip on my mac but i keep getting this error:

calibre, version 6.14.1 (darwin, embedded-python: True)
We want... a shrubbery!: Please read the friendly manual of how to install Python.

Starting job: Generating Word Wise and X-Ray for Principles
Job: "Generating Word Wise and X-Ray for Principles" failed with error:
Traceback (most recent call last):
File "calibre/gui2/threaded_jobs.py", line 82, in start_work
File "calibre_plugins.worddumb.parse_job", line 150, in do_job
install_deps("lxml", notifications)
File "calibre_plugins.worddumb.deps", line 35, in install_deps
PY_PATH, py_version = which_python()
File "calibre_plugins.worddumb.deps", line 80, in which_python
r = run_subprocess(
File "calibre_plugins.worddumb.utils", line 51, in run_subprocess
return subprocess.run(
File "subprocess.py", line 501, in run
File "subprocess.py", line 966, in init
File "subprocess.py", line 1842, in _execute_child
FileNotFoundError: [Errno 2] No such file or directory: '/opt/homebrew/bin/python3'

Called with args: ((5, 'MOBI', '/Users/emnatasha/Calibre Library/Ray Dalio/Principles (5)/Principles - Ray Dalio.mobi', <calibre.ebooks.metadata.book.base.Metadata object at 0x12604d390>, 'en'), True, True) {'notifications': <queue.Queue object at 0x12604f460>, 'abort': <threading.Event object at 0x12604f550>, 'log': <calibre.utils.logging.GUILog object at 0x12604f670>}

can someone help me with this? thank you