Large library in KOReader

[ctop]

Hi there,
I wonder how folks here are handling large libraries in KOReader. I currently have about 4500 epubs on my devices, indexed using the book scan within KOReader. Whenever I go to the filemanager to look for a new book, it takes ages (like in: several minutes) to open the first screen. On the bottom it says something like page 12 of 430 pages, so I know, there is a lot of calculation going on to put my in the right place (although that is not the place I want to be in..). So this is clearly not the best way to handle this, but what are the alternatives? BTW, my library is handled in Calibre and sent to the device, but not using collections, just a simple structure of Authorname/Title on the file system.
There must be better ways to do this!
All the best,
Ctop

[NiLuJe]

What is the FM sorting by? IIRC, some sort methods are massively slower than others.

(Also, on which device is that ).

And what, exactly, are you calling "the book scan"?

[ctop]

Originally Posted by NiLuJe

What is the FM sorting by? IIRC, some sort methods are massively slower than others.

(Also, on which device is that ).

And what, exactly, are you calling "the book scan"?

I am currently using KOReader on a Likebook Ares and a Kobo Forma. The behaviour can be observed on both devices.

The sorting is currently on "name mixed" (this takes more than 10 minutes to display the first screen on my Ares), but I have tried other sorting methods, like "last read" or "filename" to see if this speeds things up, but to no avail. Any suggestions on what might be most effective are welcome.

What I mean by book scan is the "Book info cache management", which scans all the books for metadata and puts them in a cache (if I understand correctly).

All the best,
Ctop

[ilovejedd]

Which part is slow? By file manager or by Calibre metadata search?

If it's the file manager, add some more hierarchies to the directory structure. A folder on FAT32 with thousands of entries in it doesn't make for good performance.

For example:

{author_sort:.1}/{author_sort}/{title}

If I remembered the syntax correctly, that should give you:

/L/Last Name, First Name/Title

[Frenzie]

I think filename, date added, date modified, and size should be the fastest (based on a quick peek at the code, not experiments). Mixed filenames & folders adds another step on top of regular filename, for example.

Basically you'll just want to subdivide a little further; even without speed concerns I imagine 400+ pages are very inefficient to navigate. ilovejedd posted a good suggestion for that.

[pazos]

Originally Posted by ctop

What I mean by book scan is the "Book info cache management", which scans all the books for metadata and puts them in a cache (if I understand correctly).

That's not needed. Specially if you put a huge number of documents on your reader. AFAIK it is used for the cover view. The alternative: do not index. CoverBrowser will do that for you the first time you open a folder, which hopefully will have less than 50-100 books, instead of the thousands you're trying to index at once.

FWIW, I have a bunch of pdf documents on my cervantes, stored in pdfs/work/year/month. Some months I gather 50-100 documents, other months less than 10. In any case I don't do any indexing and most of the time I do not read them on the device, but with the device hooked to a computer.

I just did the test on September 2019 folder (121 documents, most of them single page receipts but a few large memos). Never opened from the filemanager before. The filemanager opens them in "classic mode view" and start updating them one by one (1s each one aprox).

[Frenzie]

Actually CoverBrowser does it per page/view, not for the whole folder, afaik. :-)

[Ken Maltby]

I have over 7000 documents/ebooks that are on my external micro-SD card. They are in some 50+ subject/category folders, behind a .books hidden folder. My "Home" is that .books folder, it is currently three pages, at ~14 entries/folders per page.

Luck;
Ken

[NiLuJe]

Yeah, definitely don't play with the BookInfo cache, it's not doing what you think it does (like, at all) .

It's hidden seventy nesting layers deep for a reason ^^.

(The only non-developer valid use-case for messing with that is if you've massively changed your Library's content, and want to purge old thumbnails from no-longer-available books... to end up saving, like, 5MB.

TL;DR: Don't.).

[ilovejedd]

Originally Posted by NiLuJe

(The only non-developer valid use-case for messing with that is if you've massively changed your Library's content, and want to purge old thumbnails from no-longer-available books... to end up saving, like, 5MB.

Lol, I actually use it quite often to batch update metadata for in-progress fics.

For kicks, I tried copying 5,458 EPUB files to a new test folder on the PW4. Indexing took 1 hour 3 minutes.

/Fandom/A/Authors - Series 01 - Title

Most number of files in a single folder is 654. Browsing is reasonably fast with file system based sort methods although I probably should've nested it some more.

It seems when you open a "parent" folder, the "child" folders also get scanned so it can display how many items/"grandchildren" there are in the the "child" folders. That said, even with 3125 "grandchildren" inside 27 "child" folders, opening the parent folder didn't take too long (2-3 seconds).

[NiLuJe]

I'm, err, actually unclear on what you're doing to actually force a scan.

AFAICT, *nothing* forces a scan:

* Compact just runs a VACUUM on the DB
* Prune just does what it says on the tin, it queries the existing, already-scanned entries, and removes them if the file doesn't exist anymore.

:?

[ilovejedd]

Originally Posted by NiLuJe

I'm, err, actually unclear on what you're doing to actually force a scan.

AFAICT, *nothing* forces a scan:

* Compact just runs a VACUUM on the DB
* Prune just does what it says on the tin, it queries the existing, already-scanned entries, and removes them if the file doesn't exist anymore.

:?

While in the folder to be scanned, plus (+) sign on upper right corner, last entry is Extract and cache book information.

[NiLuJe]

Oh, right, I was still stuck on the OP's menu entries . Thanks .

[ctop]

Thanks for all of you for the replies. So it seems I should split my library in a range of subdirectories, which seems fair enough. The reason I did not do so from the start is that it seemed to me (please correct me if I am wrong) that the file search will cover only the current directory, without going down into the file system hierarchy. Also, when I start file search, I am only given the choice of "current folder" and "home folder". If I could add more folders here it might be easier to search as well.

All the best,

Ctop

[NiLuJe]

A file search is going to be atrociously slow no matter what you do. On the upside, it is recursive.

A Calibre metadata search *might* be faster (especially subsequent searches), or it might murder your RAM and crash on a Library of that size ^^.

(That's the "Find a book in calibre catalog" entry just above file search. It's extremely badly worded, as it has nothing to do with catalogs, and everything to do with the calibre metadata flat db that everyone gets as long as you use "Send to Device". Remind me to reword it.).