Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > KOReader

Notices

Reply
 
Thread Tools Search this Thread
Old 02-02-2020, 09:29 PM   #1
ctop
Connoisseur
ctop is on a distinguished road
 
Posts: 50
Karma: 60
Join Date: Jun 2008
Device: zaurus->palm->iPad->Sony PRS-T1,T2,T3->Kobo Forma&Likebook Ares
Large library in KOReader

Hi there,
I wonder how folks here are handling large libraries in KOReader. I currently have about 4500 epubs on my devices, indexed using the book scan within KOReader. Whenever I go to the filemanager to look for a new book, it takes ages (like in: several minutes) to open the first screen. On the bottom it says something like page 12 of 430 pages, so I know, there is a lot of calculation going on to put my in the right place (although that is not the place I want to be in..). So this is clearly not the best way to handle this, but what are the alternatives? BTW, my library is handled in Calibre and sent to the device, but not using collections, just a simple structure of Authorname/Title on the file system.
There must be better ways to do this!
All the best,
Ctop
ctop is offline   Reply With Quote
Old 02-03-2020, 12:57 AM   #2
NiLuJe
BLAM!
NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.
 
NiLuJe's Avatar
 
Posts: 9,848
Karma: 17500002
Join Date: Jun 2010
Location: Paris, France
Device: Kindle 2i, 3g, 4, 5w, (PW) & PW2; Kobo H2O & Forma
What is the FM sorting by? IIRC, some sort methods are massively slower than others.

(Also, on which device is that ).

And what, exactly, are you calling "the book scan"?
NiLuJe is offline   Reply With Quote
Advert
Old 02-03-2020, 02:13 AM   #3
ctop
Connoisseur
ctop is on a distinguished road
 
Posts: 50
Karma: 60
Join Date: Jun 2008
Device: zaurus->palm->iPad->Sony PRS-T1,T2,T3->Kobo Forma&Likebook Ares
Quote:
Originally Posted by NiLuJe View Post
What is the FM sorting by? IIRC, some sort methods are massively slower than others.

(Also, on which device is that ).

And what, exactly, are you calling "the book scan"?
I am currently using KOReader on a Likebook Ares and a Kobo Forma. The behaviour can be observed on both devices.

The sorting is currently on "name mixed" (this takes more than 10 minutes to display the first screen on my Ares), but I have tried other sorting methods, like "last read" or "filename" to see if this speeds things up, but to no avail. Any suggestions on what might be most effective are welcome.

What I mean by book scan is the "Book info cache management", which scans all the books for metadata and puts them in a cache (if I understand correctly).

All the best,
Ctop
ctop is offline   Reply With Quote
Old 02-03-2020, 02:35 AM   #4
ilovejedd
hopeless n00b
ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.
 
ilovejedd's Avatar
 
Posts: 4,549
Karma: 13300048
Join Date: Jan 2009
Location: in the middle of nowhere
Device: PW4, PW3, Libra H2O, iPad 9.7, iPad 10.5, iPad 12.9
Which part is slow? By file manager or by Calibre metadata search?

If it's the file manager, add some more hierarchies to the directory structure. A folder on FAT32 with thousands of entries in it doesn't make for good performance.

For example:

Code:
{author_sort:.1}/{author_sort}/{title}
If I remembered the syntax correctly, that should give you:

Code:
/L/Last Name, First Name/Title
ilovejedd is offline   Reply With Quote
Old 02-03-2020, 05:12 AM   #5
Frenzie
Guru
Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.
 
Posts: 917
Karma: 300000
Join Date: Oct 2014
Location: Antwerp
Device: Kobo Aura H2O
I think filename, date added, date modified, and size should be the fastest (based on a quick peek at the code, not experiments). Mixed filenames & folders adds another step on top of regular filename, for example.

Basically you'll just want to subdivide a little further; even without speed concerns I imagine 400+ pages are very inefficient to navigate. ilovejedd posted a good suggestion for that.
Frenzie is offline   Reply With Quote
Advert
Old 02-03-2020, 12:04 PM   #6
pazos
cosiñeiro
pazos ought to be getting tired of karma fortunes by now.pazos ought to be getting tired of karma fortunes by now.pazos ought to be getting tired of karma fortunes by now.pazos ought to be getting tired of karma fortunes by now.pazos ought to be getting tired of karma fortunes by now.pazos ought to be getting tired of karma fortunes by now.pazos ought to be getting tired of karma fortunes by now.pazos ought to be getting tired of karma fortunes by now.pazos ought to be getting tired of karma fortunes by now.pazos ought to be getting tired of karma fortunes by now.pazos ought to be getting tired of karma fortunes by now.
 
Posts: 369
Karma: 387867
Join Date: Apr 2014
Device: BQ Cervantes 4
Quote:
Originally Posted by ctop View Post
What I mean by book scan is the "Book info cache management", which scans all the books for metadata and puts them in a cache (if I understand correctly).
That's not needed. Specially if you put a huge number of documents on your reader. AFAIK it is used for the cover view. The alternative: do not index. CoverBrowser will do that for you the first time you open a folder, which hopefully will have less than 50-100 books, instead of the thousands you're trying to index at once.

FWIW, I have a bunch of pdf documents on my cervantes, stored in pdfs/work/year/month. Some months I gather 50-100 documents, other months less than 10. In any case I don't do any indexing and most of the time I do not read them on the device, but with the device hooked to a computer.

I just did the test on September 2019 folder (121 documents, most of them single page receipts but a few large memos). Never opened from the filemanager before. The filemanager opens them in "classic mode view" and start updating them one by one (1s each one aprox).
pazos is offline   Reply With Quote
Old 02-03-2020, 12:18 PM   #7
Frenzie
Guru
Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.
 
Posts: 917
Karma: 300000
Join Date: Oct 2014
Location: Antwerp
Device: Kobo Aura H2O
Actually CoverBrowser does it per page/view, not for the whole folder, afaik. :-)
Frenzie is offline   Reply With Quote
Old 02-03-2020, 01:01 PM   #8
Ken Maltby
Wizard
Ken Maltby ought to be getting tired of karma fortunes by now.Ken Maltby ought to be getting tired of karma fortunes by now.Ken Maltby ought to be getting tired of karma fortunes by now.Ken Maltby ought to be getting tired of karma fortunes by now.Ken Maltby ought to be getting tired of karma fortunes by now.Ken Maltby ought to be getting tired of karma fortunes by now.Ken Maltby ought to be getting tired of karma fortunes by now.Ken Maltby ought to be getting tired of karma fortunes by now.Ken Maltby ought to be getting tired of karma fortunes by now.Ken Maltby ought to be getting tired of karma fortunes by now.Ken Maltby ought to be getting tired of karma fortunes by now.
 
Ken Maltby's Avatar
 
Posts: 4,356
Karma: 6272781
Join Date: Dec 2009
Location: The Heart of Texas
Device: JetBook Lite (retired), AuraHD, PDA,
I have over 7000 documents/ebooks that are on my external micro-SD card. They are in some 50+ subject/category folders, behind a .books hidden folder. My "Home" is that .books folder, it is currently three pages, at ~14 entries/folders per page.

Luck;
Ken
Ken Maltby is offline   Reply With Quote
Old 02-03-2020, 02:32 PM   #9
NiLuJe
BLAM!
NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.
 
NiLuJe's Avatar
 
Posts: 9,848
Karma: 17500002
Join Date: Jun 2010
Location: Paris, France
Device: Kindle 2i, 3g, 4, 5w, (PW) & PW2; Kobo H2O & Forma
Yeah, definitely don't play with the BookInfo cache, it's not doing what you think it does (like, at all) .

It's hidden seventy nesting layers deep for a reason ^^.

(The only non-developer valid use-case for messing with that is if you've massively changed your Library's content, and want to purge old thumbnails from no-longer-available books... to end up saving, like, 5MB.

TL;DR: Don't.).

Last edited by NiLuJe; 02-03-2020 at 02:35 PM.
NiLuJe is offline   Reply With Quote
Old 02-03-2020, 03:13 PM   #10
ilovejedd
hopeless n00b
ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.
 
ilovejedd's Avatar
 
Posts: 4,549
Karma: 13300048
Join Date: Jan 2009
Location: in the middle of nowhere
Device: PW4, PW3, Libra H2O, iPad 9.7, iPad 10.5, iPad 12.9
Quote:
Originally Posted by NiLuJe View Post
(The only non-developer valid use-case for messing with that is if you've massively changed your Library's content, and want to purge old thumbnails from no-longer-available books... to end up saving, like, 5MB.
Lol, I actually use it quite often to batch update metadata for in-progress fics.

For kicks, I tried copying 5,458 EPUB files to a new test folder on the PW4. Indexing took 1 hour 3 minutes.

/Fandom/A/Authors - Series 01 - Title

Most number of files in a single folder is 654. Browsing is reasonably fast with file system based sort methods although I probably should've nested it some more.

It seems when you open a "parent" folder, the "child" folders also get scanned so it can display how many items/"grandchildren" there are in the the "child" folders. That said, even with 3125 "grandchildren" inside 27 "child" folders, opening the parent folder didn't take too long (2-3 seconds).
ilovejedd is offline   Reply With Quote
Old 02-03-2020, 03:26 PM   #11
NiLuJe
BLAM!
NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.
 
NiLuJe's Avatar
 
Posts: 9,848
Karma: 17500002
Join Date: Jun 2010
Location: Paris, France
Device: Kindle 2i, 3g, 4, 5w, (PW) & PW2; Kobo H2O & Forma
I'm, err, actually unclear on what you're doing to actually force a scan.

AFAICT, *nothing* forces a scan:

* Compact just runs a VACUUM on the DB
* Prune just does what it says on the tin, it queries the existing, already-scanned entries, and removes them if the file doesn't exist anymore.

:?
NiLuJe is offline   Reply With Quote
Old 02-03-2020, 05:26 PM   #12
ilovejedd
hopeless n00b
ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.
 
ilovejedd's Avatar
 
Posts: 4,549
Karma: 13300048
Join Date: Jan 2009
Location: in the middle of nowhere
Device: PW4, PW3, Libra H2O, iPad 9.7, iPad 10.5, iPad 12.9
Quote:
Originally Posted by NiLuJe View Post
I'm, err, actually unclear on what you're doing to actually force a scan.

AFAICT, *nothing* forces a scan:

* Compact just runs a VACUUM on the DB
* Prune just does what it says on the tin, it queries the existing, already-scanned entries, and removes them if the file doesn't exist anymore.

:?
While in the folder to be scanned, plus (+) sign on upper right corner, last entry is Extract and cache book information.
ilovejedd is offline   Reply With Quote
Old 02-03-2020, 05:36 PM   #13
NiLuJe
BLAM!
NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.
 
NiLuJe's Avatar
 
Posts: 9,848
Karma: 17500002
Join Date: Jun 2010
Location: Paris, France
Device: Kindle 2i, 3g, 4, 5w, (PW) & PW2; Kobo H2O & Forma
Oh, right, I was still stuck on the OP's menu entries . Thanks .
NiLuJe is offline   Reply With Quote
Old 02-03-2020, 09:51 PM   #14
ctop
Connoisseur
ctop is on a distinguished road
 
Posts: 50
Karma: 60
Join Date: Jun 2008
Device: zaurus->palm->iPad->Sony PRS-T1,T2,T3->Kobo Forma&Likebook Ares
Thanks for all of you for the replies. So it seems I should split my library in a range of subdirectories, which seems fair enough. The reason I did not do so from the start is that it seemed to me (please correct me if I am wrong) that the file search will cover only the current directory, without going down into the file system hierarchy. Also, when I start file search, I am only given the choice of "current folder" and "home folder". If I could add more folders here it might be easier to search as well.

All the best,

Ctop
ctop is offline   Reply With Quote
Old 02-03-2020, 10:22 PM   #15
NiLuJe
BLAM!
NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.
 
NiLuJe's Avatar
 
Posts: 9,848
Karma: 17500002
Join Date: Jun 2010
Location: Paris, France
Device: Kindle 2i, 3g, 4, 5w, (PW) & PW2; Kobo H2O & Forma
A file search is going to be atrociously slow no matter what you do. On the upside, it is recursive.

A Calibre metadata search *might* be faster (especially subsequent searches), or it might murder your RAM and crash on a Library of that size ^^.

(That's the "Find a book in calibre catalog" entry just above file search. It's extremely badly worded, as it has nothing to do with catalogs, and everything to do with the calibre metadata flat db that everyone gets as long as you use "Send to Device". Remind me to reword it.).

Last edited by NiLuJe; 02-03-2020 at 10:34 PM.
NiLuJe is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
KOReader doesn't open large, complex ebooks? SbAgOoTs KOReader 33 01-13-2020 01:40 PM
KoReader folder bug see on Nickel library anki Kobo Reader 4 12-28-2019 08:24 PM
How large is your Calibre library? calibrejunkie Calibre 64 10-07-2019 11:25 PM
Creating a dedicated library partition for koreader. Markismus Kobo Developer's Corner 6 09-02-2013 10:02 AM
Best way to get a large Calibre library into the PE library? Filark enTourage Archive 0 04-20-2011 11:18 PM


All times are GMT -4. The time now is 03:46 AM.


MobileRead.com is a privately owned, operated and funded community.