dictutil: Tools, documentation, and libraries related to Kobo dictionaries

[davidfor]

Originally Posted by loyukfai

Wish Kobo will revert the change, so no need to do this manual action every time the firmware is updated...

You mean all of once since this change was made? Yep, such a pain.

In case you don't recognise it, there was a lot of sarcasm in that statement.

[loyukfai]

Originally Posted by davidfor

You mean all of once since this change was made? Yep, such a pain.

In case you don't recognise it, there was a lot of sarcasm in that statement.

I don't really get it... mind to explain?

Cheers.

[davidfor]

Originally Posted by loyukfai

Thanks a lot! Done and working.

Wish Kobo will revert the change, so no need to do this manual action every time the firmware is updated...

Cheers

Originally Posted by loyukfai

I don't really get it... mind to explain?

Reread your original post. Now, how many times have you had to to this "manual action"? Once so far. And unless something goes wrong, once every three months or so at the most. Add in the fact that Kobo does not support sideloading dictionaries, so anything you do needs some fiddling and probably needs to be redone for each firmware. Sorry, but, I just can't find any sympathy for the complaint.

[loyukfai]

Hahaha, ok, thanks for the explanation.

Cheers.

Admin/Mods: Feel free to delete my posts to clean up the thread. Thx.

[Alanon]

I've been trying to get back to making some dictionaries after a long time, and I'm more than slightly confused by all these new tools. Previously I think I used a combination of makedict (which I can no longer find?) and/or Penelope, and had success converting dictionaries that I still use. Some were StarDict, but some were converted through at least one more format before I was able to get them Kobo-ready.

Anyway, there are newer dictionaries I'd like to try out, mostly in .dsl and .mdx formats, and I'd hoped these new tools would help me along. So I started from scratch and set up PyGlossary and dictutil, with the intention of using the glossary to convert my source files to some cleaner format, and then using dictutil to do the rest.

Despite all my efforts, I was not able to make a successful conversion once with these new tools, so I must be doing something wrong. During the course of my research, i realised that there are different versions of .mdx, and that revision 3 is not really convertable, so I ditched the format for the time being and focused on dsl.

Using the default PyGlossary options to convert my dictionaries to Kobo format did not result in a readable dictionary. So I guess my first question would be - to what format do I need to convert a .dsl dictionary for best results with dictutils?

[geek1011]

Originally Posted by Alanon

Using the default PyGlossary options to convert my dictionaries to Kobo format did not result in a readable dictionary. So I guess my first question would be - to what format do I need to convert a .dsl dictionary for best results with dictutils?

If you don't have words with multiple headwords (i.e. multiple words with the same definition) and don't have images, I'd actually suggest using Penelope to convert the dictionary, then to use dictzip-decompile on it and regenerate it with dictgen. This way, you can take advantage of the input formats supported by Penelope, but still make use of the features and compatibility of dictutil.

Otherwise, if you have multiple headwords, I'd suggest converting it to a plain-text format (like the Babylon text one or CSV) and using a text editor with regexp and multi-cursor support to transform it into a dictfile, then use dictgen on it. If you do use the Babylon format, there's a converter on the dictutil site.

If you also need images, I'd suggest writing a custom script to do it (or give me an example of what a word looks like, and I'll see what I can do).

[Semwize]

Originally Posted by Alanon

I used a combination of makedict (which I can no longer find?)

makedict

But I would recommend using PyGlossary, makedict is deprecated and abandoned by the author.

Originally Posted by Alanon

to what format do I need to convert a .dsl dictionary for best results with dictutils?

(lsd -> dsl) dsl -> StarDict -> Penelope -> dictzip-decompile

[Alanon]

Thank you both very much! Now I know that I'm doing something wrong!

First, I tried to convert a .dsl file into a StarDict file and then use Penelope, which produced an invalid Kobo dictionary, and compile/recompile with dictutil didn't help. Then I tried an old StarDict file I had and used Penelope again, and that worked. When I tried that same StarDict dictionary and used PyGlossary directly to get a Kobo dictionary, I got a write error:

Spoiler:

Using Reader class from Stardict plugin for direct conversion without loading into memory

Writing to Kobo format requires full sort, falling back to indirect mode
Loaded 91968 entries
Writing to file 'D:\\Downloads\\Portable Python 3.6.5 x64\\AHD4-2.4.2\\AHD4.kobo'
Exception while calling plugin's write function
Traceback (most recent call last):
  File "D:\Downloads\Portable Python 3.6.5 x64\pyglossary-3.3.0\pyglossary\glossary.py", line 932, in write
    writer.write(filename, **options)
  File "D:\Downloads\Portable Python 3.6.5 x64\pyglossary-3.3.0\pyglossary\plugins\ebook_kobo.py", line 97, in write
    words = self.write_groups(group_by_prefix_length)
  File "D:\Downloads\Portable Python 3.6.5 x64\pyglossary-3.3.0\pyglossary\plugins\ebook_kobo.py", line 80, in write_groups
    with gzip.open(group_fname + ".html", mode="wb") as gzipFile:
  File "D:\Downloads\Portable Python 3.6.5 x64\App\Python\lib\gzip.py", line 53, in open
    binary_file = GzipFile(filename, gz_mode, compresslevel)
  File "D:\Downloads\Portable Python 3.6.5 x64\App\Python\lib\gzip.py", line 163, in __init__
    fileobj = self.myfileobj = builtins.open(filename, mode or 'rb')
OSError: [Errno 22] Invalid argument: 'B<.html'


Writing file 'D:/Downloads/Portable Python 3.6.5 x64/AHD4-2.4.2/AHD4.kobo' failed.

I understand that Kobo is a fresh addition to PyGlossary version 3.3, so it could just be that the kinks haven't yet been worked out. Finally, I tried makedict (thank you Semwize!) to convert a .dsl to StarDict and used Penelope to get to Kobo, which worked. Decompile and compile with dictutil worked also. (Btw, are there any recommended parameters for compile/recompile of Penelope dictionaries? While I was testing, I just used drag and drop. I suppose that my old conversions would benefit from a dictutil pass-through, so if it's simple enough I might just go through them all.)

I have yet to try and tinker with Babylon or any other features that depend on PyGlossary. I was able to find a different tool that extracts the content from .mdx files and exports it directly to a .txt file, but one look into the file tells me that removing all the stylesheet references, internal links, span tags and the like is going to be a huge pain. Still, it seems to be the best option for .mdx files to date, if I can only streamline the process of cleaning up the file so that dictutil can read it. (I'd appreciate any tips on that.)

Bottom line is, PyGlossary seems to be the point of error for me, as it somehow corrupts the StarDict version and that poisons the rest of the chain. The thing is, there are no errors during the conversion process from .dsl to StarDict, as far as I can see, but if one thing isn't working, it stands to reason that others might not work either. I can't seem to pinpoint what I'm doing wrong. I'd certainly like to get PyGlossary working since most people consider it to be the norm, if nothing else than to try and extract some .mdx data from it and see if it's any easier than the method I've found.

[Semwize]

Originally Posted by Alanon

I'd certainly like to get PyGlossary working since most people consider it to be the norm

Who consider this the norm)) I then edit the code manually (with regular expressions, as geek1011 advised above), because I don’t like what PyGlossary creates (indentation, etc.). Just makedict is really out of date.

[geek1011]

Originally Posted by Alanon

I understand that Kobo is a fresh addition to PyGlossary version 3.3, so it could just be that the kinks haven't yet been worked out. Finally, I tried makedict (thank you Semwize!) to convert a .dsl to StarDict and used Penelope to get to Kobo, which worked.

I just looked through the code for PyGlossary, and it's not going to output correctly for anything but the simplest dictionaries. It also doesn't support images, multiple definitions, multiple headwords, cyrillic words, maybe not accented words, and words with numbers/spaces/symbols.

If there's enough interest, though, I might make a dictfile output for PyGlossary, which can then be used with dictutil.

Decompile and compile with dictutil worked also. (Btw, are there any recommended parameters for compile/recompile of Penelope dictionaries? While I was testing, I just used drag and drop. I suppose that my old conversions would benefit from a dictutil pass-through, so if it's simple enough I might just go through them all.)

Yes, it's pretty simple. In 99% of cases, you just need to do `dictzip-decompile whatever.zip`, then `dictgen whatever.df`. You'll also be able to edit and merge the dictfiles (remember that duplicate entries are fine).

[Semwize]

geek1011, stardict. I once tried to convince you that we, with Cyril and Methodius quite a lot, you did not believe

[Alanon]

Originally Posted by geek1011

I just looked through the code for PyGlossary, and it's not going to output correctly for anything but the simplest dictionaries. It also doesn't support images, multiple definitions, multiple headwords, cyrillic words, maybe not accented words, and words with numbers/spaces/symbols.

If there's enough interest, though, I might make a dictfile output for PyGlossary, which can then be used with dictutil.

Ok, that makes sense. I tried to play around some more, but no matter what I tried, I wasn't able to get PyGlossary to output a readable dictionary. I'd hoped that it might prove useful for reading .mdx and as a tool for automatic conversions, but at this point I'm just about ready to give up on it.

However, makedict with Penelope still works as it should, and your tool also has tremendous value. Keeping with my standard conversion steps (dsl to stardict to kobo), I was able to make some tweaks thanks to the decompiler, and improve the overall readability of my conversions. I'm still battling with multiple blockquotes and ways to disentangle them, and learning how to use regex replacement features, but I can already see what a difference it can make!

I did encounter an interesting issue that I presume has to do with Kobo rather than my dictionaries. Many glyphs used in English phonetic transcription seem to be missing? Is this an encoding error or are they actually missing from the default font? (For example, "o͞o" is rendered as two o's with a block above them.)

[jackie_w]

Originally Posted by Alanon

I did encounter an interesting issue that I presume has to do with Kobo rather than my dictionaries. Many glyphs used in English phonetic transcription seem to be missing? Is this an encoding error or are they actually missing from the default font? (For example, "o͞o" is rendered as two o's with a block above them.)

Kobo's built-in Georgia may (???) be adequate for the built-in dictionaries but if you're sideloading custom dictionaries you'll probably need something better.

Which font is "better" will vary depending on those dictionaries. Currently there are 2 workarounds:

• Sideload the "better" font and use the `Dictionary text font-family/font-size/line-height` patch in the kobopatch system to specify this better font.
• Give a facelift to the whole Kobo GUI by replacing the built-in Georgia with the "better" font (internally renamed to masquerade as Georgia).

ETA: In case it helps, the char you quoted, "o͞o", is present in Arial. This is what I use in for my dictionary widgets using option 1. above.

[Alanon]

Originally Posted by jackie_w

Kobo's built-in Georgia may (???) be adequate for the built-in dictionaries but if you're sideloading custom dictionaries you'll probably need something better.

Which font is "better" will vary depending on those dictionaries. Currently there are 2 workarounds:

• Sideload the "better" font and use the `Dictionary text font-family/font-size/line-height` patch in the kobopatch system to specify this better font.
• Give a facelift to the whole Kobo GUI by replacing the built-in Georgia with the "better" font (internally renamed to masquerade as Georgia).

ETA: In case it helps, the char you quoted, "o͞o", is present in Arial. This is what I use in for my dictionary widgets using option 1. above.

GREAT tip! I am actually using this patch but completely forgot about its font editing capabilities. This really helped with the rendition and overall style of the definitions. I already had a modified version of Linux Libertine on the device, so I used that, as it should have more than enough glyphs for anyone's needs. So far it's picked up on all the exotic glyphs. I highly recommend it, it's actually a beautiful and useable typeface. Slowly but surely, I'm getting better and better results with my tinkering. I'll probably be redoing all my conversions now.

[MGlitch]

Ok, running firmeware 4.22.15190 the trick of setting the dictionary files to read only does not seem to work.

At least on my Forma a sync is replacing them with the ones they're meant to stand in for. Specifically "dicthtml-fr-en" and the "dicthtml" (which for me is english, but this is how it's displayed on the Kobo).


I've tried setting it to read only on both Windows 10 and OS X Catalina. Both respect the others settings for this so I know it's being properly preserved by Kobo. But once I sync the files are replaced.

edit to add, I've also tried, in terminal, chmod 444 on the files in question and the dict folder itself. Same result as above the files are replaced.
I'm not sure if this change also affects this tool, as quite honestly I've no idea how to use it. But I presume it does as I'm guessing it's just setting the dicthtml files to read only.

Any thoughts?