KOReader: a document reader for PDF, DJVU, EPUB, FB2, HTML, ... (GPLv3)

[sadowski]

Originally Posted by hawhill

I guess the way would be fiddling with epub.css, indeed. We just took what came with CoolReader Engine, I think. It's a bit messy and there simply isn't a solution that fits all cases, unfortunately.

That said, CREngine does not properly understand all CSS properties either. It works surprisingly well, though.

My impression is that margin-top after a page break is simply ignored. Other than in ADE, this is even so if the new page starts at the beginning of a new xml document within the epub.

[guerrilla]

Calibre viewer and kindle stock reader show the top margin if the item is wrapped inside a div tag.
Unfortunately it turns out it's a known bug in CREngine
http://sourceforge.net/p/crengine/bugs/157/

[guerrilla]

Ok, now I'm fighting with dictionary. As I can't find a decent italian dictionary for stardict, I figured it would be nice to take a wiktionary.org dump and convert it to stardict format.

Yet it seems I can't produce a working (in koreader) dictionary. Well, at a certain point it seems I produced a working dictionary, but I can't figure out what went right that time and what is going constantly wrong since then.

I tried everything: utf-8 with dom, utf-8 without dom, not using dos CR, using dos CR, tab format, babylon source format. I'm compiling the source file with stardict editor for windows. The dictionary is always fine with stardict for pc, but it doesn't work in koreader (both stable and nightly build).
Koreader's log says:

# lookup word: test
# stripped word: test
# JSON data cannot be decoded frontend/JSON.lua:525: frontend/JSON.lua:240: can't parse JSON at char 1 of: save to cache data/dict/test.idx
[{
		"dict":	"test",
		"word":	"test",
		"definition":	"\ndictionary works"
	}]
# dummy result table: test {
	[1] = {
		["dict"] = "",
		["word"] = "test",
		["definition"] = "Nessuna definizione trovata."
	}
}
# showing quick lookup window
# show widget
# close widget

Dictionary files are enclosed so that maybe somebody can tell me what I'm doing wrong.


UPDATE

I uncommented in readerdictionary.lua the line that gives you log info about results_str from sdcv and I compared the log with those from a working entry. It seems that every time JSON.lua fails to decode the result string it's because sdcv added the substring "save to cache data/dict/test.idx" to it. Could that be the reason? (in which case, line 592 in sdcv/blob/master/src/lib/lib.cpp should be the one causing the issue).

Below is the log from the not working entry in the enclosed dictionary. Logs from the working dictionary I mentioned above don't show the string "save to cache data/dict/test.idx" in the results.

# result str: test save to cache data/dict/test.idx
[{
		"dict":	"test",
		"word":	"test",
		"definition":	"\ndictionary works"
	}]

[knc1]

Where is the pathname fragment: data/dict/test.idx being written to on the PC and on the Kindle?
(I.E: what is the full pathname in each case, working and non-working?)
Also check for differences in the permissions on both.

- - - - - -

Have you inspected the lines in the JSON.lua error report?

# JSON data cannot be decoded frontend/JSON.lua:525: frontend/JSON.lua:240: can't parse JSON at char 1 of: save to cache data/dict/test.idx

(I.E: perhaps 'save to cache' is not implemented in JSON.lua)

Ah...
Are you using the same JSON.lua on the Kindle as on the PC?

[guerrilla]

No wait knc1, when I wrote that the dictionary is working on the pc I meant with stardict for windows, not koreader for pc. I haven't access to a linux install at the moment, so I can't compile a pc build of koreader.

As for JSON.lua, as far as I can tell, it actually doesn't manage that "save to cache" output. But is JSON.lua used to parse only output from stcv? Otherwise I would catch that string in readerdictionary.lua or just comment line 592 in lib.cpp; is it of any use for sdcv to print that to stout?
But maybe I'm getting everything wrong, I only know some very basic stuff of procedural programming and absolutely nothing specific to lua or cpp...

[knc1]

Originally Posted by guerrilla

No wait knc1, when I wrote that the dictionary is working on the pc I meant with stardict for windows, not koreader for pc. I haven't access to a linux install at the moment, so I can't compile a pc build of koreader.

If you are not using the same JSON decoder/encoder library on both systems, aren't you trying to compare the taste of apples and oranges?

Ah, *.lua files are plain text, just read those two lines referenced in the error report, in context.
(It is Lua or LuaJIT that does the compiling, at run-time.)

[guerrilla]

Originally Posted by knc1

If you are not using the same JSON decoder/encoder library on both systems, aren't you trying to compare the taste of apples and oranges?

I meant to say that, if the dictionary works in stardict for windows, maybe the issue is not related to the dictionary files.

Originally Posted by knc1

Ah, *.lua files are plain text, just read those two lines referenced in the error report, in context.
(It is Lua or LuaJIT that does the compiling, at run-time.)

Yeah I know, I think that what I wrote in the previous two posts (please read them carefully) clearly implies that I went through (at least) readerdictionary.lua.
Those two lines, as far as I can tell, don't add much to the conclusions I came to, but again I'm no real programmer.
They basically write the message you see in the logs I posted, when encountering a decode error.

[knc1]

So try editing your dictionary file(s) to remove the lines the JSON decoder claims it can not parse.

All you **might** lose is look-up caching -- which if we can believe the messages, you don't have anyway.

[barobach_so]

hi
sorry it may be long story but be patient please
as it said here koreader can be customized by stardict glossaries
but, I have a problem with dictionary in Persian language

when I try to create stardict glossaries from other formats ( bgl,... ) and copy them to .kobo/koreader/data/dict they don't work and nothing is seen.

I tried different input glossary and now have BGL bgl xml csv html dict dic ifo idx epub prc ... glossary and when convert them to stardict format no one work in koreader

I tried penelope, Marisa and Pyglossary ( GUI and command propmt method ) in Windows and even dictconv in Ubuntu and no one make a change in my problem and still no ifo work and nothing is seen.

finaly I found stardict glossary of my language online by chance and when I copy it to device it shows some square shape

then I change my strategy to add some font to koreader
I add every UTF font I found ( include Noto Arabic ) in Fonts folder in root and font folder in koreader directory

and then change font defaults in fonts.lua file. change cfont tfont and fallback font

and replace my noto font with the name of original one on kobo and replace them

finally I get here :



its left to right ( wrong orientation ) and seprate character

can anybody help me with one of these:

make a good ifo glossary

make this working by chance glossary to show well

[guerrilla]

Originally Posted by guerrilla

It seems that every time JSON.lua fails to decode the result string it's because sdcv added the substring "save to cache data/dict/test.idx" to it. Could that be the reason? (in which case, line 592 in sdcv/blob/master/src/lib/lib.cpp should be the one causing the issue).

So it's confirmed that sdcv is causing this issue by printing to stdout the string: "save to cache" and the path of the relevant idx file.
The string is then passed to JSON.decode which fails to decode it as it probably assumes that the first char of the input string should be [.
I worked around the issue with a wrapper for sdcv with this code:

#!/bin/sh
./sdcv "$@" | /bin/sed -e "s/^save to cache .*\.idx$//"

and I changed line 65 in readerdictionary.lua to run the wrapper instead of sdcv.

Turns out that the italian dictionary I already have is not that bad at all and I was under that impression because koreader was giving dummy results for a lot of queries due to this issue.
What I find odd is that nobody noticed this so far. It's maybe because sdcv usually does this cache thing only the first time you look for a term. Sometimes though it gets stuck caching the same term over and over and people probably assume that the term is not in the dictionary?

If this is confirmed as a bug, I guess that the ultimate solution would be to clean results_str with a string substitution before passing it to JSON.decode at line 72 of readerdictionary.lua.

[Doitsu]

Originally Posted by barobach_so

koreader can be customized by stardict glossaries but, I have a problem with dictionary in Persian language

AFAIK, KOReader uses the Coolreader engine, which doesn't support RTL languages and the developer stated that he won't implement RTL support. I.e. there's nothing you can do about it.
(If you test your dictionary with the desktop versions of StarDict or GoldenDict, it'll most likely work.)

[hawhill]

For the dictionary, we even render ourselves in a most basic fashion - which is not aware of RTL (besides other things it is not aware of). Once I planned to use CoolReader for rendering the dictionary output (in order to render HTML tags in there), but as Doitsu wrote, that wouldn't bring RTL support either.

@guerrilla: Very good find. Care to do a report an issue on github if you haven't yet? Though I think that fixing sdcv would be the way to go - instead of hacking around the issue. Arguably, that (diagnostics) output should go to stderr instead of stdout.

[guerrilla]

@hawhill: That would be even better and was actually my first guess, but I was unsure if you were willing to fork sdcv (or maybe it's already forked?).
The patch would be very simple, as far as I can tell it's just about changing line 592 of sdcv/src/lib.cpp from:

		printf("save to cache %s\n", url.c_str());

to:

		fprintf(stderr,"save to cache %s\n", url.c_str());

right?

I'm still unsure about how I got stuck in this though. At a closer look it seems that that message is printed only when sdcv creates the cache file (.oft extension) and this should be the first time you search for a term. That's why this issue got unnoticed so far: you install your dictionary and search for a term, sdcv gives the "save to cache" message, json.decode fails to decode, koreader returns the dummy table, you think: "uh, word not present in the dictionary" and make another search which, from now on, goes smoothly.
For some reason I was getting that warning again and again.

Anyway, I will gladly do a report to github, I'm very happy to help

[barobach_so]

Originally Posted by Doitsu

AFAIK, KOReader uses the Coolreader engine, which doesn't support RTL languages and the developer stated that he won't implement RTL support. I.e. there's nothing you can do about it.
(If you test your dictionary with the desktop versions of StarDict or GoldenDict, it'll most likely work.)

thank you for the reply
but how koreader has a RTL and LTR option in reflow engine ? when I read a pdf with koreader I can change this option

and other question:
there is a glossary.BGL that say it is graphical and no need font ( I don't know really waht is it ), can it be somehow helpful ?

and the last Question:
u know anything about ArabShaping ( or here ) ? can it solve RTL problem? there is some patch for android for RTL problem in xda-developers
I'm thinking of change bgl to html (babylontohtml program ) and have a html glossary ( already have ) then revers the characters somehow and maybe connect them to each other ( arabic is a cursive font )
there is a prc glossary for kindle with the method I say and build by mobipocket

[hawhill]

@guerrilla: we already forked sdcv anyway :-) Thank you very much for the report!

@barobach_so: The PDF reflow engine isn't based on text rendering of any kind: It takes a visual approach and works on a rendered image of a page in the PDF, reordering what it finds in there. As in: There's a bunch of pixels that constitute a word (with some probability), and a few words in a row are a line.

The dictionary doesn't do HTML. I planned to implement it at some point, but simply do not have any spare time for the coming months.

This is not android software. Also, it doesn't use libICU. There is no sophisticated text layout engine. Given the amount of work it takes to implement one or even integrate an existing one, don't expect this problem to be solved soon. There aren't that many developers and there time is, unfortunately, scarce. A friendly invitation goes to anyone who wants to help, though. But mainly, this means writing code.