KOReader crashs when trying to search Calibre metadata

[Uncle Robin]

After finishing a book 20 minutes ago, I decided to "Browse series" for my next book. KOReader crashed. I tried again, it crashed again. So I tried "Browse tags" and it crashed again, tried "browse tags" again, and it crashed again. That' probably why the crash log was too big to attach, hence this Google Drive link - sorry!

[hius07]

Check for the nightly build
https://github.com/koreader/koreader/issues/9085

[pazos]

Originally Posted by hius07

Check for the nightly build
https://github.com/koreader/koreader/issues/9085

There're different issues. The other one was due to a refactor in timestamps across the whole project and was easy to fix. This one... I have no idea.

Seems something buried into the UI stack. We need to wait for our setDirty expert @NiLuJe

[NiLuJe]

Umm, I'm only seeing the crash fixed by #9085 in there? Did I miss one?

[Uncle Robin]

Thanks all - the latest nightly has fixed it.

[pazos]

Huh?. I didn't expect that from that stacktrace.

Anyways. Happy to see it is already fixed

[b00kwyrm]

Sorry to necropost, but I am having this problem too on 2024.04. Where can I find logs to troubleshoot or otherwise share?

[pazos]

Originally Posted by b00kwyrm

Sorry to necropost, but I am having this problem too on 2024.04. Where can I find logs to troubleshoot or otherwise share?

https://github.com/koreader/koreader.../bug_report.md

[mathil]

This has been happening to me with my Kobo Libra 2. My database is, admittedly, quite large (390mb after compression with the KoboUtilities plugin, almost 4000 books on the device). I've been checking github, specifically the latest related bug report here, and am I right in understanding that this issue cannot be solved?

[pazos]

Originally Posted by mathil

This has been happening to me with my Kobo Libra 2. My database is, admittedly, quite large (390mb after compression with the KoboUtilities plugin, almost 4000 books on the device). I've been checking github, specifically the latest related bug report here, and am I right in understanding that this issue cannot be solved?

Everything can be solved

No idea what's your issue. Follow the bug report template and attach logs (here or in github).

Most likely your calibre json file size is above 30MB and the program switches to a utterly broken mode where it tries to parse json manually, instead of using a proper parser.

That's because the "proper parser" - rapidjson- with its current lua bindings doesn't provide a way to build the result in steps. It tries to build everything in memory until the result is ready.

So the hardcap of 30MiB is there to prevent users with <512MiB or even with +512MiB and high memory pressure to get an out of memory error.

The proper solution is to implement a parser to use for big files, but in the mean time tweaking the hardcap https://github.com/koreader/koreader...ent-1101370638 will work up to the size of your available ram.

[pazos]

Another related issue is that, unless you push books using wireless transfers, we consume calibre files as is, with a bunch of metadata we don't care and we don't use.

In wireless transfers we just discard these fields before writting the file, so the json ends up being much more slim.

The difference isn't too much for calibre beginners (up to 10x) but grows a lot if you're one of those advanced users with toons of custom metadata.

[mathil]

Thank you for your help, pazos - everything is much clearer now. I had already found that link before actually and tried to tweak the hardcap but without understanding what I was doing

My metadata.calibre file is 39.3MB, and changing the max_json_filesize to 40 * etc. doesn't work. The Kobo Libra 2's RAM is 512MB, so I'm not sure it's wise to try and increase that further. And you are correct in assuming that I'm an 'advanced' Calibre user Right now, I'm not willing to change that (or how I send books to the device - I'm still using the Kobo original firmware alongside koreader) so I'm not sure there's much to be done. I'll keep using the regular file search in the meantime.

I would be willing to share the log if you think that's helpful.

[pazos]

Make sure those 39.3 are in fact MB and not MiB because that's more than 40x1000x1000 bytes

It should be safe to raise the size to 50 x 1000 x 1000. Do that, restart the program and try again. One of the following two things will happen:

1. The program crashes and you see "./luajit: not enough memory"
2. Everything works.

I hope it is the second, please let me know

[mathil]

Originally Posted by pazos

Make sure those 39.3 are in fact MB and not MiB because that's more than 40x1000x1000 bytes

It's 39.379KB (as shown on Windows), so you're right, more than 40MB. I had to Google that... I think I learned something new today LOL

I did try with 50 x 1000 x 1000 and it works perfectly now. Thank you for your patience

[pazos]

Originally Posted by mathil

It's 39.379KB (as shown on Windows), so you're right, more than 40MB. I had to Google that... I think I learned something new today LOL

I did try with 50 x 1000 x 1000 and it works perfectly now. Thank you for your patience

Great! Keep in mind you'll need to reapply the change after each update.

Hopefully you won't need to do it many times as a proper fix is in the works: https://github.com/koreader/koreader/pull/11922