dictutil: Tools, documentation, and libraries related to Kobo dictionaries

geek1011 · 03-05-2020, 05:38 PM

dictutil
Tools, documentation, and libraries related to Kobo dictionaries (and a few converted ones).
___

This project contains a collection of tools and libraries to work with Kobo dictionaries, plus comprehensive documentation of Kobo's dictionary format.

Unlike previous attempts at working with Kobo dictionaries, dictutil has full support for all features supported by nickel (word prefixes, unicode, variants, images, etc), with a focus on simplicity, correctness (prefix generation and other features are directly tested against libnickel's code and regexps, v1/v2 dictionaries are differentiated), and completeness (most of the research was done by reverse-engineering libnickel).

In addition, it has a custom format for creating Kobo dictionaries which has a simple syntax and full support for all features.

Dictutil consists of multiple tools and libraries:

dictutil provides commands for installing, removing, unpacking, packing, and performing low-level modifications and tests on Kobo dictionaries. All operations are intended to be correct, lossless, and deterministic.
dictgen simplifies creating full-featured dictionaries for Kobo eReaders, with support for images, unicode prefixes, raw html, markdown, and more.
dicthtml documents Kobo's dictionary format and how it works.
examples/gotdict-convert is a working example of using dictutil to convert GOTDict into a Kobo dictionary.
examples/webster1913-convert is a working example of using dictutil to convert Project Gutenberg's Webster's Unabridged Dictionary into a Kobo dictionary.
examples/dictzip-decompile is an experimental tool to convert a dictzip into a dictfile.
examples/bgl-convert is a simple tool to convert Babylon BGL dictionaries to a dictfile.
Library: kobodict provides support for reading, writing, encrypting, and decrypting Kobo dictionaries.
Library: dictgen provides the functionality of dictgen as a library.
Library: marisa provides a simplified self-contained CGO wrapper for marisa-trie.

Dictutil implements version 2 of the Kobo dictionary format, which supports firmware versions 4.7.10364+.

See the website for more details and examples.

Quick reference:

dictgen:

Spoiler:

dictgen dictfile format:

Spoiler:

dictutil:

Spoiler:

dictutil install:

Spoiler:

dictutil uninstall:

Spoiler:

dictutil pack:

Spoiler:

dictutil unpack:

Spoiler:

dictutil prefix:

Spoiler:

gotdict-convert:

Spoiler:

webster1913-convert:

Spoiler:

dictzip-decompile:

Spoiler:

Download | Website

geek1011 · 03-05-2020, 05:38 PM

Dictionaries

This post contains a pre-built dictionaries converted using the tools in examples/*-convert. Links to the source dictfiles are also included for use as examples, for merging with your own dictionaries, or for editing.

Webster's 1913 Unabridged Dictionary (from gutenberg.org/ebooks/29765)

Dictionary (4.7.10364+): dicthtml-wb.zip (source: webster1913.df)

GOTDict (from github.com/wjdp/gotdict)

Images (4.20.14601+): dicthtml-gt.zip (source: gotdict.df)
No images (4.7.10364+): dicthtml-gt.noimg.zip (source: gotdict.noimg.df)

Generated by dictutil v0.3.1.

geek1011 · 03-05-2020, 05:39 PM

Changelogs

v0.3.1:

Spoiler:

v0.3.0:

Spoiler:

v0.2.1:

Spoiler:

v0.2.0:

Spoiler:

v0.1.2:

Spoiler:

v0.1.1:

Spoiler:

v0.1.0:

Spoiler:

geek1011 · 03-05-2020, 05:40 PM

reserved

geek1011 · 03-06-2020, 12:36 AM

I've released v0.1.1 with some small fixes to the converted Webster's dictionary, and a fix for segfaults from marisa on Windows, and v0.1.2 with a fix for file permissions on Windows. Thanks to Semwize for reporting the issues on Windows!

Semwize · 03-06-2020, 02:03 AM

Works

It would be great to add stardict converter (xml) -> df
and very good - the ability to convert dicthtml -> df

Cyfranek · 03-06-2020, 04:14 PM

@geek1011 please tell me:
- can I add my custom dictionary (Polish) using dicutil?
dictutil install dicthtml_pl.zip

- can I add my custom translation dictionary (English-Polish) using dicutil?
dictutil install dicthtml_en-pl.zip

Polish is not supported by Kobo, so I don't know if it's possible with Your tool. If not, can I replace other dictionary (Portugeese for instance) by Polish ones and prevent them from syncing "oryginal" one? How?
TIA

Semwize · 03-06-2020, 04:41 PM

Quote:

Originally Posted by Cyfranek

- can I add my custom dictionary (Polish) using dicutil?
dictutil install dicthtml_pl.zip

Yes, the program will write to sqlite and place the dictionary in the desired Kobo folder .

Quote:

Originally Posted by Cyfranek

- can I add my custom translation dictionary (English-Polish) using dicutil?
dictutil install dicthtml_en-pl.zip

No, rename it, for example dicthtml_e1.zip

geek1011 · 03-06-2020, 05:16 PM

Semwize's answers are correct.

Quote:

Originally Posted by Cyfranek

Polish is not supported by Kobo, so I don't know if it's possible with Your tool. If not, can I replace other dictionary (Portugeese for instance) by Polish ones and prevent them from syncing "oryginal" one? How?

You can replace any built-in locale, and dictutil will automatically prevent syncing the original one.

__
P.S. Semwize: Thanks for all the help with testing dictutil (I'd give you karma, but I have to spread it around first)!

geek1011 · 03-06-2020, 05:21 PM

Quote:

Originally Posted by Semwize

It would be great to add stardict converter (xml) -> df

I've decided against that one, as dictutil is not intended to be a general-purpose dictionary tool (that's what Penelope is for), and because the stardict format is too broad (there are 4+ different body text formats I would have to implement from scratch).

Quote:

and very good - the ability to convert dicthtml -> df

I'm working on it right now, and am nearly finished.

It will support seamless decompilation of dictionaries generated by Penelope, Kobo (the unencrypted ones, if any), or dictgen (although it won't be able to recover the original Markdown or images, only the raw HTML). Dictionaries generated by other tools or manually created will also be able to decompiled, but it won't be able to automatically extract the header and generate the best dictfile.

It will also be useful for fixing bugs with prefixes, variants, and missing words in existing dictionaries (just decompile and regenerate the dictfile). Another use will be merging dictionaries (just decompile it, and merge the resulting dictfiles).

You will also be able to use it to convert stardict dictionaries by using Penelope to convert it to a dictzip, then this tool to decompile it to a dictfile, and then modify it from there.

JSWolf · 03-06-2020, 06:23 PM

Will dictutil allow converting a Kindle format dictionary to a Kobo format dictionary?

geek1011 · 03-06-2020, 06:55 PM

Quote:

Originally Posted by JSWolf

Will dictutil allow converting a Kindle format dictionary to a Kobo format dictionary?

No, as that would require implementing a MOBI parser and a whole lot more code. Besides, that's starting to get outside the scope of dictutil. You can use another tool to convert it to a usable format, then from there to a dictfile, though.

geek1011 · 03-06-2020, 09:12 PM

Quote:

Originally Posted by Semwize

and very good - the ability to convert dicthtml -> df

I've released v0.2.0 with a new dictzip-decompile tool. See the documentation for more information.

In addition, I've increased the maximum line buffer size for dictgen, which fixes compiling some large dictfiles with long lines of generated HTML.

Semwize · 03-07-2020, 04:48 AM

Here is what I will note. When creating Extra dictionaries with the help of Penelope, did not found (found, but no definition was given) in the dictionary phrases ('word word', 'word word word').

I creating the dictionary using dictutil and now everything is fine.

Quote:

Originally Posted by geek1011

I've decided against that one

Yes, it is superfluous in essence.

geek1011 · 03-07-2020, 08:40 AM

Quote:

Originally Posted by Semwize

Here is what I will note. When creating Extra dictionaries with the help of Penelope, did not found (found, but no definition was given) in the dictionary phrases ('word word', 'word word word').

I creating the dictionary using dictutil and now everything is fine.

For the record, you will also note the same thing for words with accented characters. Under certain conditions, you'll also see the same for words starting with Cyrillic characters or containing non-ascii characters after the first two non-whitespace ones. In addition, this is more of a theoretical case, but you'll also see the same for words starting with whitespace.

See here and here.

03-05-2020, 05:38 PM	#2
geek1011 Wizard Posts: 2,735 Karma: 6990705 Join Date: May 2016 Location: Ontario, Canada Device: Kobo Mini, Aura Edition 2 v1, Clara HD	Dictionaries This post contains a pre-built dictionaries converted using the tools in examples/-convert. Links to the source dictfiles are also included for use as examples, for merging with your own dictionaries, or for editing. Webster's 1913 Unabridged Dictionary* (from gutenberg.org/ebooks/29765) Dictionary (4.7.10364+): dicthtml-wb.zip (source: webster1913.df) GOTDict (from github.com/wjdp/gotdict) Images (4.20.14601+): dicthtml-gt.zip (source: gotdict.df) No images (4.7.10364+): dicthtml-gt.noimg.zip (source: gotdict.noimg.df) Generated by dictutil v0.3.1. Last edited by geek1011; 07-09-2020 at 06:18 PM. Reason: v0.3.1

03-05-2020, 05:39 PM	#3
geek1011 Wizard Posts: 2,735 Karma: 6990705 Join Date: May 2016 Location: Ontario, Canada Device: Kobo Mini, Aura Edition 2 v1, Clara HD	Changelogs v0.3.1: Spoiler: Patrick Gaskin (@geek1011) a4519b2 Checked built-in dictionaries against 4.21.15015 4e955b7 examples/webster1913-convert: Fixed end license being included as a word 5b406d0 docs: Updated dictionary sync patch for 4.22.15190 53c3fc5 Checked built-in dictionaries against 4.22.15190 fdc0799 Added support for installing translation dictionaries (closes #10) v0.3.0: Spoiler: 6c333b4 all: Switched windows builds to 64-bit e442434 docs: Added links to pre-built dictionaries to the main header 50aa051 docs: Clarified the current situation with read-only dictionaries (#6) 939235f cmd/dictgen: Added option to prevent the footer from showing on certain locales (fixes #5) 8ef75c2 dictgen: Added PostRawHTML option to DictFileEntry 10e0680 all: Added integration tests for overall dictzip-related functionality 1a820d5 examples/webster1913-convert: Fixed false positives for headwords dae2dd1 marisa: Fixed reading large tries, added debug env var 5bfa4c5 marisa: Finished with the last bit of cleanup 0239665 all: Make marisa optional for building kobodict fdc6c66 marisa: Fixed a few subtle errors and simplified the stream shim 018a9ae docs: Updated links a84722e docs: Added documentation for examples/bgl-convert f490dfc examples/bgl-convert: Added new tool to convert Babylon BGL dictionaries 1a80ab6 marisa: Cleaned up everything 0b4392a marisa: Cleaned up imports 65ab5e6 all: Added Travis builds for testing on macOS bbb361d marisa: Added support for directly using io.Reader/io.Writer 3930b08 all: Removed SWIG from AppVeyor builds 338b88a marisa: Fixed tests on non-amd64 architectures c91db49 marisa: Rewrote wrapper to run completely in-memory and not depend on SWIG 3c56b2e kobodict: Abstracted marisa-trie bindings into an interface 61364e6 all: Checked built-in dictionaries against 4.20.14622 bca4c53 examples/webster1913-convert: Handle synonyms starting after the example of a numbered definition 0b8179b cmd/dictutil: Fixed missing newline in fmt.Printf v0.2.1: Spoiler: ec6e0d7 all: Don't upload converted dictionaries from Drone 6913bda examples/webster1913-convert: Cleaned up leading and trailing whitespace in parsed dictionary b6f76da cmd/dictutil: Added warning about dictionary label not having an effect on 4.20.14601+ 96b86db all: Updated dictionary installation for 4.20.14601 v0.2.0: Spoiler: 8a0e9ee all: Updated release build script ccfb5f6 examples/dictzip-decompile: Implemented a tool to convert a dictzip into a dictfile 1e207d6 dictgen: Increased maximum line buffer size v0.1.2: Spoiler: 7e56ba0 cmd/dictutil: Fixed file permissions on Windows v0.1.1: Spoiler: a761abd all: Fixed AppVeyor builds 652765b marisa: Removed pre-generated SWIG wrapper for now (fixes #2) 78994bb docs: Added MobileRead link to nav 0881c03 docs: Fixed link on homepage c9e9847 examples/webster1913-convert: Fixed phrase definitions v0.1.0: Spoiler: 6024447 all: Updated dependencies 025a074 all: Fixed AppVeyor artifact download links 8dca7a7 all: Use AppVeyor to build pre-built dictionaries for examples/* 54d1ad1 examples/webster1913-convert: Separated parser into its own package 1aedd40 examples/gotdict-convert: Separated parser into its own package e49fcfd all: Added package and command godoc comments 53ccb4e examples/webster1913-convert: Also add phrase definitions as variants so they can be looked up 555d4f1 examples/webster1913-convert: Added new tool for parsing and converting Gutenberg's Webster's 1913 dictionary 2e5270c docs: Improved nav order 1635ed0 docs: Finshed documentation for dictutil commands f08f72b docs: Added getting started section to main page 164094b docs: Improved dicthtml format examples 382b790 docs: Finished dictgen and dictfile documentation 45c48a8 cmd/dictgen: Improved wording of help text ad77e8c docs: Added note about custom dictionary labels on 4.20.14601 e984c78 cmd/dictutil: Added note about dictionary labels on 4.20.14601 712d5fa docs: Finished prefix documentation 6492cbc dictgen: Updated notes about image methods (#1) 10eadad docs: Updated gotdict-convert docs da6d4ad docs: Updated dictgen docs 01da35f examples/gotdict-convert: Prefer the version with images (#1) d05535b cmd/dictgen: Enable images with base64 encoding by default (#1) 512c333 docs: Updated for 4.20.14601 2a80ab3 all: Updated Go to 1.14 5931c7f examples/gotdict-convert: Made version string consistent with other dictutil commands da99419 examples/gotdict-convert: Updated gotdict, removed workarounds 8de70b8 kobodict: Further simplified and improved performance of WordPrefix bc52ded marisa: Removed compile-time dependency on SWIG 7cdb1d8 kobodict: Fixed memory leak from marisa 7276923 examples/gotdict-convert: Fixed build 3533800 all: Updated build dependency graph 641efd5 all: Disabled image support by default (#1) b403da2 dictgen: Added Description func to ImageHandler aa108b2 cmd/dictgen: Show more information about options 66d311f dictgen: Extracted ImageFunc from WriteDictZip for extensibility b3acaf0 all: Added README, updated links in help text 2ccdbb6 dictgen: Added some doc comments 1aeb44b dictgen: Added support for image CSS, fixed images displaying too small b56c1b8 examples/gotdict-convert: Implemented image support a739fca cmd/dictgen: Implemented image support 46a4fd3 dictgen: Implemented image support 744fca8 kobodict: Added Exists function on Writer c7a1fc8 cmd/dictutil: Fixed locale removal when ExtraLocales contains spaces 3d3008f docs: Misc fixes 54ab4a1 docs: Added stubs for all pages, wrote some more docs ce7bc60 docs: Finished format docs e1e4b25 cmd/dictutil: Implemented prefix command ccc6c72 examples/gotdict-convert: Improved Drone config for uploads 2f4c068 kobodict: Improved parameter name 55fa1c6 all: Added version info to commands dea5f2e all: Cleaned up old files, updated some TODOs 0ac4192 cmd/dictutil: Implemented uninstall command 0e4ba01 cmd/dictutil: Made help text more complete in Drone config f2e29b8 cmd/dictutil: Implemented install command c24c2fb marisa: Silenced the -Wimplicit-fallthrough warning 70ace33 docs: Added alternative dictionary installation method d5bd3b7 all: Added cross-compilation and releases to Drone config 25e0621 cmd/dictutil: Changed return code to 0 for help text 6edaf98 docs: Fixed site URL 79969ed docs: Changed Jekyll theme to just-the-docs 932448c docs: Added info about installing dictionaries fcb478e cmd/dictutil: Implemented dictutil command 99aedd6 dictgen: Added test to ensure the reparsing the generated dictfile is identical to the original 95ceef9 examples/gotdict-convert: Also generate and upload dictzip in Drone config 0551eb2 dictgen: Implemented Validate 2865b7c kobodict: Implemented Pack and Unpack 5539d29 all: Updated TODOs eaf7d0c kobodict: Added stubs for Pack/Unpack 0dbd59c kobodict: Implemented Reader 7f22617 kobodict: Fixed build 384d955 cmd/dictgen: Implemented encryption e89d5b5 dictgen: Improved API by allowing access to the kobodict.Writer 514a77c kobodict: Implemented Crypter, fixed bugs in Writer ccf7f38 all: Updated Drone config 2927cb3 cmd/dictgen: Implemented initial version of the dictgen CLI fccdcbb dictgen: Implemented dictzip generation 2d3b54c kobodict: Implemented kobodict.Writer abda141 examples/gotdict-convert: Fixed definition image cleanup 0cc2589 cmd/dictutil: Reorganized files 2462a54 examples/gotdict-convert: Misc CLI improvements 3befc43 examples/gotdict-convert: Fixed CLI arguments b54757e all: Added license 461c408 dictutil: Updated main func ac9f819 docs: Added dictword-test and marisa to index 958390a all: Made Drone config more deterministic 33ff613 docs: Updated index 60664a6 examples/gotdict-convert: Implemented tool to convert github.com/wjdp/gotdict 91a6e55 marisa: Included merged libmarisa sources instead of depending on the lib. 81dbf10 all: Disable cross-compiling marisa for macOS for now 9b653e9 all: Cross-compile marisa ce0107c all: Fixed Drone config 3ea1df4 marisa: Added Go wrapper 725206f all: Simplified Drone config 226f40a dictgen: Implemented parser and dicthtml generation ec22c1a kobodict: Cleaned up outdated TODOs c12cbf1 all: Ignore GH Pages build in Drone config 8512fa8 kobodict: Finished implementing reverse-engineered word prefix and normalization stuff d5297b8 docs: Added note about kobo-mods/dictword-test 3758fcf docs: Added some notes about v1/v2 dictionaries and stubs for other docs 5a5a443 docs: Initial commit Last edited by geek1011; 07-08-2020 at 04:13 PM. Reason: v0.3.1

03-05-2020, 05:40 PM	#4
geek1011 Wizard Posts: 2,735 Karma: 6990705 Join Date: May 2016 Location: Ontario, Canada Device: Kobo Mini, Aura Edition 2 v1, Clara HD	reserved Last edited by geek1011; 03-07-2020 at 08:42 AM.

03-06-2020, 12:36 AM	#5
geek1011 Wizard Posts: 2,735 Karma: 6990705 Join Date: May 2016 Location: Ontario, Canada Device: Kobo Mini, Aura Edition 2 v1, Clara HD	I've released v0.1.1 with some small fixes to the converted Webster's dictionary, and a fix for segfaults from marisa on Windows, and v0.1.2 with a fix for file permissions on Windows. Thanks to Semwize for reporting the issues on Windows! Last edited by geek1011; 03-06-2020 at 01:53 AM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Adding new dictionaries to kobo.	Robik	Kobo Developer's Corner	61	12-19-2023 04:30 PM
Useful Related Tools Threads	BetterRed	Related Tools	4	01-28-2022 05:03 PM
Mini Sync Problems on 3 different Kobo Minis (related to software update?)	jpottle	Kobo Reader	2	06-20-2013 12:57 PM
Copy/Move books between libraries using command line tools	jameszh	Library Management	3	02-15-2011 09:02 AM
MSD (BBeB dictionaries) tools ?	Papi	LRF	0	12-20-2010 04:07 AM

03-06-2020, 02:03 AM	#6
Semwize Guru Posts: 873 Karma: 252902 Join Date: Jun 2016 Device: Kobo	Works It would be great to add stardict converter (xml) -> df and very good - the ability to convert dicthtml -> df

03-06-2020, 04:14 PM	#7
Cyfranek Groupie Posts: 163 Karma: 35750 Join Date: Jul 2017 Location: Poland Device: PocketBok Era, Kindle Oasis 3, Bookeen Diva HD	@geek1011 please tell me: - can I add my custom dictionary (Polish) using dicutil? dictutil install dicthtml_pl.zip - can I add my custom translation dictionary (English-Polish) using dicutil? dictutil install dicthtml_en-pl.zip Polish is not supported by Kobo, so I don't know if it's possible with Your tool. If not, can I replace other dictionary (Portugeese for instance) by Polish ones and prevent them from syncing "oryginal" one? How? TIA

03-06-2020, 06:23 PM	#11
JSWolf Resident Curmudgeon Posts: 73,970 Karma: 128903378 Join Date: Nov 2006 Location: Roslindale, Massachusetts Device: Kobo Libra 2, Kobo Aura H2O, PRS-650, PRS-T1, nook STR, PW3	Will dictutil allow converting a Kindle format dictionary to a Kobo format dictionary?