[GUI Plugin] WordDumb

[ArthurQ]

Originally Posted by xxyzz

BTW, calibre can create MOBI news directly. You need to set 'preferred output format' to MOBI in preferences->behavior.

Thank you, now I can create MOBI news .
And after I tried to dis-checked 'columns have three values(needs restart)' option, WordDumb works for me

[xxyzz]

I added images to X-Ray for MOBI and AZW3 books in commit https://github.com/xxyzz/WordDumb/co...1888f4ef1b5f95, the latest build can be downloaded from GitHub Actions(https://github.com/xxyzz/WordDumb/ac...uns/1596064865). Please try it out and report bugs.

For KFX books, I'll need jhowell's help again. The X-Ray data for KFX looks like this:

sqlite> SELECT * FROM excerpt WHERE image NOT NULL;
+-----+---------+--------+-------+------------------+---------+
| id  |  start  | length | image | related_entities |  goto   |
+-----+---------+--------+-------+------------------+---------+
| 576 | 1       | 0      | e6    |                  | 1       |
| 577 | 2179585 | 71     | e2GX  |                  | 2179583 |
| 578 | 2179667 | 57     | e2H8  |                  | 2179665 |

The "goto" column is the image tag offset in the file, I guess the "image" will be similar to the "src" of the img tag in MOBI file.

The "start" and "length" are for the sentence below the image. For convenience(laziness), I set "start" to the same as "goto" and "length" to 0 in this commit.

[jhowell]

Originally Posted by xxyzz

For KFX books, I'll need jhowell's help again. The X-Ray data for KFX looks like this:

sqlite> SELECT * FROM excerpt WHERE image NOT NULL;
+-----+---------+--------+-------+------------------+---------+
| id  |  start  | length | image | related_entities |  goto   |
+-----+---------+--------+-------+------------------+---------+
| 576 | 1       | 0      | e6    |                  | 1       |
| 577 | 2179585 | 71     | e2GX  |                  | 2179583 |
| 578 | 2179667 | 57     | e2H8  |                  | 2179665 |

It never occurred to me that images would be relevant to the X-Ray feature. I would be happy to help.

In that table the image field has values starting with "e". Those are identifiers for KFX resources.

The code I wrote to extract text and positions from KFX files currently handles only text. I can enhance that to also include the resource names and positions for images. Each image in KFX occupies one "position" within the book, the same as a single character.

If you need to access the actual image content then I can also include the mapping from resource name to image location. That is the filename of the image within the unpacked book. (You can unpack KFX using a different option of the KFX Input plugin.)

Is there anything else you might need?

[yiming]

Thank you for creating this great plugin. I've have used it on a few books from the Standard Ebooks and found that soft hyphens are still causing problem. For eg, in Pride and Prejudice, the word "property" is treated as "prop", and in The Great Gatsby, the word "wagon" is treated as "wag".

Could you look into the problem. I'm using the latest 3.14.4 of the plugin. Thanks.

links to the books:
https://standardebooks.org/ebooks/ja...prejudice.azw3
https://standardebooks.org/ebooks/f-...at-gatsby.azw3

[xxyzz]

Originally Posted by jhowell

It never occurred to me that images would be relevant to the X-Ray feature.

There is an image tab in X-Ray shows all the images in the book, it's quite convenient to find images.

Originally Posted by jhowell

Is there anything else you might need?

The image resource name and position are sufficient for the task. Thanks for your help.

[xxyzz]

Originally Posted by yiming

that soft hyphens are still causing problem

Both libraries(flashtext and spaCy) I used for creating Word Wise and X-Ray can't deal with soft hyphen. You have to remove soft hyphens or convert the book to KFX with KFX Output plugin.

[jhowell]

Originally Posted by xxyzz

The image resource name and position are sufficient for the task. Thanks for your help.

No problem. I will let you know when it is ready.

[yiming]

Originally Posted by xxyzz

Both libraries(flashtext and spaCy) I used for creating Word Wise and X-Ray can't deal with soft hyphen. You have to remove soft hyphens or convert the book to KFX with KFX Output plugin.

Thanks for your reply. I misunderstood this: "Update to v3.7.5, this version ignores soft hyphens in the book." Thought that it applies to AZW3 as well.

[xxyzz]

Originally Posted by yiming

I misunderstood this: "Update to v3.7.5, this version ignores soft hyphens in the book." Thought that it applies to AZW3 as well.

That version does ignore soft hyphen when creating Word Wise but not X-Ray. And since v3.13.0 I'm using flashtext to improve speed, so soft hyphen affects Word Wise again.

[jhowell]

Originally Posted by jhowell

No problem. I will let you know when it is ready.

Version 1.46 of the KFX Input plugin has the change to include position information for images. See the section on "Generating content/position information" in the KFX Input documentation for a description of how the output format has changed.

Let me know if there are any problems.

[yiming]

Originally Posted by xxyzz

That version does ignore soft hyphen when creating Word Wise but not X-Ray. And since v3.13.0 I'm using flashtext to improve speed, so soft hyphen affects Word Wise again.

Thanks for the clarification.

[xxyzz]

v3.15.0

- update dependencies
- close MOBI file
- support X-Ray images, thanks for jhowell's help

[mmobes]

So when the instructions say the plugin won't support macOS, does that mean I can't use macOS to add X-Ray to my books through Calibre or does that mean X-Ray won't work when reading on macOS?

[xxyzz]

Originally Posted by mmobes

So when the instructions say the plugin won't support macOS, does that mean I can't use macOS to add X-Ray to my books through Calibre or does that mean X-Ray won't work when reading on macOS?

It means you can't use this plugin to create X-Ray file on the official calibre macOS app. Use WordDumb on Linux if you want to try that feature or build calibre from source and disable library validation.

[xxyzz]

v3.16.0

- X-Ray can be created on macOS
- add 5038 and update 608 lemmas
- update dependencies